JP5975791B2 - Image processing apparatus and method, and image display apparatus and method - Google Patents

Description

  The present invention relates to an image processing apparatus and method, and an image display apparatus and method. The present invention particularly relates to a frame interpolation process for inserting a new interpolation frame between frames of an image.

  A hold-type display such as a liquid crystal display continues to display the same image for one frame period, and when the object in the image moves, the tracking of the human eye to the moving object moves continuously, There is a problem that the edge portion looks blurry because the object moves discontinuously in units of one frame. On the other hand, it is conceivable to increase the number of display frames by interpolating frames to make the object move smoothly.

  In addition, for a material in which a film image such as a movie is converted into a TV signal, two or three frames are image signals made from the same frame because of the difference in frame frequency between the two (film image and TV signal). However, if displayed as it is, there is a problem that the motion is blurred or judder with a jerky motion occurs.

  Similarly, for a material in which computer-processed video is converted into a television signal, two frames are image signals made from the same frame, and there is a problem that judder occurs similarly when displayed as it is.

  The conventional image processing apparatus and method are either a zero-order hold method in which interpolation is performed with the same image as the previous frame with respect to the interpolation frame, or an average image of the previous image and the subsequent image with respect to the interpolation frame. However, since the zero-order hold method does not move smoothly with respect to an image moving in a certain direction, the problem of blurring of the hold type display is still not solved. Further, the average value interpolation method has a problem that a moving image becomes a double image.

  As an improvement measure, interpolation is performed from the pixel having the highest correlation between the pixels on the temporally previous frame and the temporally subsequent frame that are in point symmetry with respect to the interpolated pixel of the interpolated frame. Some generate interpolated pixels of a frame (see, for example, Patent Document 1). In this method, since correlation is detected in units of pixels, it may be detected that the correlation between pixels is large even though the contents of the image are different, and an interpolation frame may not be generated correctly.

Japanese Patent Laying-Open No. 2006-129181 (page 8, FIG. 3)

  The conventional frame interpolation process is configured as described above, and there is a problem that motion is blurred or judder with motion is generated. In addition, in the method of detecting the correlation in pixel units, the interpolation frame cannot be generated correctly because the correlation cannot be detected correctly, and particularly when the object is hidden or appears before and after the frame to be interpolated, the interpolation frame is significantly disturbed. In addition, the frame interpolation processing has a problem that the calculation amount is large (the circuit scale is large in hardware).

The image processing apparatus of the present invention
In an image processing apparatus for inserting an interpolation frame between a current frame of an image and a first delay frame which is a frame one frame before the current frame,
The first movement from the second delay frame to the first delay frame with reference to the data of the first delay frame and the data of the second delay frame which is a frame two frames before the current frame A motion vector detector that calculates a vector and calculates a second motion vector from the current frame to the first delay frame by referring to the data of the current frame and the data of the first delay frame;
The first motion vector is converted into a third motion vector from the first delay frame to the interpolation frame, and the second motion vector is converted into a fourth motion vector from the current frame to the interpolation frame. A motion vector conversion unit to convert,
The interpolation frame data is generated from the third motion vector, the fourth motion vector, the first delay frame data, and the current frame data, and the interpolation frame data is converted into the current frame data. An interpolation frame generator for outputting image data inserted between the data and the data of the first delay frame,
The motion vector detection unit
The current frame is reduced by shifting the phase at random for each pixel, and the current reduced frame is output. The first delayed frame is reduced by shifting the phase at every pixel, and the first reduced delay frame is output. A dither reduction unit that outputs the second reduced delay frame by reducing the phase of the second delayed frame at random for each pixel, and
And a motion vector expanding unit that expands and outputs a motion vector detected from the current reduced frame, the first reduced delay frame, and the second reduced delay frame.

  According to the present invention, the center frame of three consecutive frames in time is most likely to be the most probable by evaluating the motion vector from the previous frame and the subsequent frame to the center frame in time. Since an accurate motion vector can be calculated, an interpolation frame can be interpolated without image distortion. Furthermore, even when an object is hidden or appears before and after the interpolated frame, the interpolated frame can be interpolated without image distortion. In addition, by using a reduced image for motion vector detection, the amount of computation (circuit scale) can be reduced. Furthermore, by performing the image reduction process by randomly shifting the phase, it is possible to suppress a decrease in detection accuracy of the motion vector due to the use of the reduced image.

It is a block diagram which shows the structure of the image display apparatus which concerns on embodiment of this invention. (A) And (b) is a figure which shows arrangement | positioning of the pixel in the reduced image produced | generated by normal image reduction and dither reduction. It is a block diagram which shows the specific example of the test interpolation part 4 in the motion vector detection part 12 of FIG. It is a block diagram which shows the specific example of the interpolation data evaluation part 5 and the motion vector determination part 6 in the motion vector 2 of FIG. It is a figure for demonstrating operation | movement of the motion vector detection part 12 of FIG. It is a figure for demonstrating operation | movement of the motion vector conversion part 13 and the interpolation frame production | generation part 14 of FIG. It is a figure for demonstrating operation | movement of the motion vector conversion part 13 and the interpolation frame production | generation part 14 of FIG. It is a figure for demonstrating operation | movement of the motion vector conversion part 13 and the interpolation frame production | generation part 14 of FIG. It is a figure which shows the example of the image in each of the present frame and the 1st and 2nd delay frame. It is a figure which shows the specific example of the image data which reduced the input image shown in FIG. It is a figure which shows two pairs of blocks located in the 2nd reduction delay frame SF2 and the current reduction | decrease frame SF0 which are a point symmetrical position centering | focusing on the process target block on 1st reduction delay frame SF1. It is a figure which shows the production | generation of the test interpolation data based on a pair of data of 2 pairs of blocks of FIG. It is a figure for demonstrating the specific example of operation | movement of the 1 motion vector detection part. It is a figure which shows the production | generation of the test interpolation data based on another pair of data of 2 pairs of blocks of FIG. It is a figure which shows the production | generation of the test interpolation data based on the data of one block on the present reduction frame SF0. It is a figure which shows the production | generation of the test interpolation data based on the data of another one block on the present reduction | decrease frame SF0. It is a figure which shows the production | generation of the test interpolation data based on the data of one block on 2nd reduction delay frame SF2. It is a figure which shows the production | generation of the test interpolation data based on the data of another one block on 2nd reduction delay frame SF2. It is a figure for demonstrating the specific example of operation | movement of the motion vector expansion part 7 of FIG. (A) And (b) is a figure for demonstrating the specific example of operation | movement of the motion vector conversion part 13 of FIG. It is a figure for demonstrating the specific example of operation | movement of the interpolation frame production | generation part 14 of FIG. (A) And (b) is a figure which shows the example of the pattern of the change of the pixel value in an original image and a reduction image. (A) And (b) is a figure which shows the example of the pattern of the change of the pixel value in the original image and reduced image compared with the image of Fig.21 (a) and (b). 1 is a block diagram illustrating an image processing apparatus configured with a computer system used when image processing according to an embodiment of the present invention is implemented by software. It is a flowchart which shows the process of the image processing apparatus of FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The image processing apparatus according to the present invention generates a new interpolated frame by interpolation between a current frame in a frame sequence constituting an input moving image and a frame one frame before the current frame. The image display apparatus according to the present invention includes an image display unit that displays the image data output from the image processing apparatus.

  FIG. 1 is a diagram showing a configuration of an image display apparatus according to an embodiment of the present invention. The illustrated image display device includes an image processing device 10 and an image display unit 15. The image processing device 10 includes a frame memory 11, a motion vector detection unit 12, a motion vector conversion unit 13, and an interpolation frame generation unit 14. And an image display unit 15.

  This image processing apparatus can constitute a part of, for example, a broadcast receiving apparatus, an image recording / reproducing apparatus, or a personal computer (PC). Further, the image display unit 15 can constitute a part of, for example, a television, a video projector, or a PC.

The image data F0 is input to the frame memory 11, the motion vector detection unit 12, and the interpolation frame generation unit 14.
The frame memory 11 stores image data F0 representing the current frame image of the input moving image for two frames, image data F1 delayed by one frame with respect to the image data F0, and two frames for the image data F0. The delayed image data F2 is output. Thus, the image data F0 is referred to as current frame data, the image data F1 is referred to as first delay frame data, and the image data F2 is referred to as second delay frame data. The current frame, the first delay frame, and the second delay frame are represented by the same symbols F0, F1, and F2 as the respective frame data.

  The data of the first delay frame F1 is input to the motion vector detection unit 12 and the interpolation frame generation unit 14, and the data of the second delay frame F2 is input to the motion vector detection unit 12.

  The motion vector detection unit 12 refers to the data of the current frame F0, the data of the first delay frame F1, and the data of the second delay frame F2, and each of the plurality of blocks on the first delay frame F1. (Consisting of a plurality of pixels that form part of a frame) are sequentially selected, and for the selected block, the first motion vector MV1 from the second delay frame F2 to the first delay frame F1 and A second motion vector MV2 from the current frame F0 to the first delay frame F1 is calculated and output to the motion vector conversion unit 13. The block on the first delay frame F1 is obtained by dividing the first delay frame F1 into, for example, a plurality of rectangular areas.

  The motion vector conversion unit 13 converts the first motion vector MV1 and the second motion vector MV2, the third motion vector MV3 from the first delay frame F1 to the interpolation frame IF, and the current frame F0 to the interpolation frame IF. Is converted into the fourth motion vector MV4 and output to the interpolated frame generation unit 14.

The interpolated frame generation unit 14 determines the interval between the current frame F0 and the first delay frame F1 from the data of the first delay frame F1, the data of the current frame F0, the third motion vector MV3, and the fourth motion vector MV4. Image data DO composed of a frame sequence in which the data of the interpolated frame IF is generated and the data of the generated interpolated frame IF is inserted between the data of the current frame F0 and the data of the first delay frame F1 is displayed. To the unit 15.
The image display unit 15 displays the image data DO.

Next, the configuration of the motion vector detection unit 12 will be described in detail.
The motion vector detection unit 12 includes a dither reduction unit 2, a block cutout unit 3, a test interpolation unit 4, an interpolation data evaluation unit 5, a motion vector determination unit 6, and a motion vector enlargement unit 7. The reduction unit 2 includes a current frame dither reduction unit 21, a first delay frame dither reduction unit 22, and a second delay frame dither reduction unit 23. The block cutout unit 3 includes a current frame block cutout unit 31. And a first delayed frame block cutout unit 32 and a second delayed frame block cutout unit 33.

  The current frame dither reduction unit 21 reduces the image of the current frame F0 and outputs a current reduction frame SF0, and the first delay frame dither reduction unit 22 reduces the image of the first delay frame F1 to reduce the first delay frame F1. The reduced delay frame SF1 is output, and the second delay frame dither reduction unit 23 reduces the image of the second delay frame F2 and outputs the second reduced delay frame SF2. The reduction ratios in the current frame dither reduction unit 21, the first delay frame dither reduction unit 22, and the second delay frame dither reduction unit 23 are the same, and are represented by the symbol α.

The dither reduction in each of the dither reduction units 21, 22, and 23 will be described. First, description will be made assuming that the reduction is performed only in one dimension.
When the image is reduced to ¼ in the one-dimensional direction, it is generally divided into a group of four pixels arranged in the reduction direction. That is, as indicated by a dotted line Ga, a pixel group is formed for every four pixels close to each other. Then, a simple average value or a bilaterally weighted average value of the pixels of each group is calculated, and calculated at the center of the group of the above four pixels (the position indicated as phase PH2 in FIG. 2A). An image in which the average value pixels are arranged is defined as a reduced image.

Here, the phase indicates a position in each period on the image when a space occupied by four pixels constituting each group is regarded as one period in the original image. In the illustrated example, the phases PH0, PH1, PH2, and PH3 are each located at the center between the pixels of the four pixels in each group in the original image. In the reduced image, one pixel constitutes one cycle, and therefore the intervals between the phases PH0, PH1, PH2, and PH3 are ¼ of the inter-pixel distance of the reduced image.
In the example shown in FIG. 2A, pixels having an average value are arranged at the same phase PH2 in any of the groups of four pixels.

  On the other hand, in dither reduction, a position where the phase is randomly shifted for each pixel to be output, for example, one of pixels corresponding to phases 0 to 3 is output as a pixel of the reduced image. As a result of randomly shifting the position, for example, as shown in FIG. 2B, in the group Ga (1), in the phase PH1, in the group Ga (3), in the phase PH0, in the group Ga (3), in the phase PH2, respectively. Pixels having the average of the group of are arranged. Accordingly, in the reduced image, the distance between the pixels is not constant and varies in the range from 1 to 7. Here, the inter-pixel distance is represented by a value normalized by the pixel pitch of the original image.

  When the reduction is performed two-dimensionally, a combination of random phases is obtained in both the horizontal and vertical directions. That is, the image is divided into a plurality of rectangular areas each composed of pixels located in the horizontal direction and the vertical direction, and pixels having an average value of the pixels in each rectangular area are displayed in both the horizontal direction and the vertical direction. The pixels arranged at the positions whose phases are randomly shifted in FIG.

  In FIG. 2B, pixels having an average value obtained for each group of four pixels are arranged in any of the four phases, but the number of pixels in each group (to obtain the average) The number of phases in which the pixels on the reduced image having the average value are arranged may be larger than the number of pixels used in (1). Thereby, the pixels on the reduced image have more various phase arrangement patterns. The same applies when performing reduction in two dimensions.

  The current frame block cutout unit 31, the first delay frame block cutout unit 32, and the second delay frame block cutout unit 33 are respectively a current reduced frame SF0, a first reduced delay frame SF1, and a second reduced delay. In response to the frame SF2, each block that forms a part of the screen is cut out (extracted), and a set of pixel data (pixel values) in the block is output as block data. That is, the current frame block cutout unit 31 cuts out a block from the current reduced frame SF0, the first delay frame block cutout unit 32 cuts out a block from the first reduced delay frame SF1, and the second delay frame block cutout unit 33 cuts out a block from the second reduced delay frame SF2.

  Each block is composed of a rectangular area having a size of, for example, X pixels in the horizontal direction and Y pixels (Y lines) in the vertical direction. That is, the block cut out from the current reduced frame SF0, the block cut out from the first reduced delay frame SF1, and the block cut out from the second reduced delay frame SF2 have a vertical size (number of pixels) and a horizontal direction. The size (number of pixels or number of lines) is equal to each other.

Hereinafter, a process for generating one block in the interpolation frame IF by interpolation will be described. For this processing, the block (interpolation target block) to be interpolated in the interpolation frame IF corresponds to the block (reduced block) reduced at the same reduction ratio α as that when the reduced frames SF0, SF1, and SF2 are generated. One block F1Ba in the first reduced delay frame SF1 is set as a processing target block, the processing target block F1Ba, a plurality of blocks in the current reduced frame SF0, and a second reduced delay frame SF2 A plurality of blocks are cut out (extracted). The block cut out from the current reduced frame SF0 and the block cut out from the second reduced delay frame SF2 are in point-symmetrical positions around the processing target block F1Ba (strictly, the center), and these are Used as a pair.
The current frame block cutout unit 31 and the second delayed frame block cutout unit 33 are located symmetrically with respect to the processing target block F1Ba, one is located in the current reduced frame SF0, and the other is the second A plurality of pairs of blocks located in the reduced delay frame SF2 are cut out.

  For example, the block F0B1 (strictly the center) cut out from the current reduced frame SF0 and the block F2B1 (strictly the center) cut out from the second reduced delay frame SF2 are the first reduced delay frame. The processing target block F1Ba in SF1 (strictly, its center) is located at a point-symmetrical position, and two blocks at these point-symmetrical positions are used as a pair.

  The block pairs cut out from the current reduced frame SF0 and the second reduced delay frame SF2 correspond to the motion vector candidates detected by the motion vector detection unit 12, for example, all of the motion vector search ranges within the motion vector search range. The block is cut out. For example, when searching the range of ± HS pixels in the horizontal direction and ± VS pixels (± VS lines) in the vertical direction around the center of one block in the first reduced delay frame SF1, the second reduction delay From the frame SF2 and the current reduced frame SF0, (2HS + 1) × (2VS + 1) blocks are cut out.

  In addition, when it is not necessary to evaluate all the blocks in the search range, for example, when the range of the motion direction can be predicted in advance or by other information, only the blocks in the predicted range of the search range are included. It is good also as cutting out. Alternatively, the blocks in the search range may be cut out while thinning out (for example, every other pixel in the horizontal direction and the vertical direction).

  In the following, the number of blocks cut out from the current reduced frame SF0 and the second reduced delay frame SF2 is M, and the blocks cut out from the current reduced frame SF0 are the first to Mth blocks F0B1 to F0BM, the second reduced delay. The blocks cut out from the frame SF2 are referred to as first to Mth blocks F2B1 to F2BM. In addition, the data of each block is represented by the same code as the block.

The m-th block F2Bm (m = 1 to M) in the second reduced delay frame SF2 and the m-th block F0Bm in the current reduced frame SF0 are the processing target block F1Ba (in the first reduced delay frame SF1). Strictly speaking, since it is in a point-symmetrical position with respect to the center pixel), the horizontal shift of the block F2Bm with respect to the processing target block F1Ba is h (h = −HS to + HS), and the vertical shift is v (v = −VS to + VS), the horizontal shift of the block F0Bm with respect to the processing target block F1Ba is −h, and the vertical shift is −v.

The current frame block cutout unit 31 cuts out a plurality of, that is, first to Mth blocks in the current reduced frame, and outputs first to Mth block data F0B1 to F0BM.
The first delay frame block cutout unit 32 cuts out the processing target block F1Ba in the first reduced delay frame. This block F1Ba corresponds to a block obtained by reducing the interpolation target block in the interpolation frame IF.
The second delay frame block cutout unit 33 cuts out a plurality of, that is, the first to Mth blocks in the second reduced delay frame, and outputs the first to Mth block data F2B1 to F2BM.

  The block data of the current reduced frame SF0 and the block data of the second reduced delay frame SF2 are input to the test interpolation unit 4. The test interpolation unit 4 includes the block data of the second reduced delay frame SF2 and the block data of the current reduced frame SF0 in the second reduced delay frame SF2 that are symmetrical with respect to the processing target block F1Ba. Test interpolation data TDA1 to TDAM are generated based on the data of a block pair consisting of these blocks and the blocks in the current reduced frame SF0. A plurality of test interpolation data is generated based on the plurality of block pairs. This test interpolation is performed on the assumption that the position of the center of point symmetry, that is, the data of the processing target block F1Ba is unknown. The more accurate the interpolation, the more the test interpolation data is processed. The correlation with the data of the block F1Ba is strong.

The test interpolation unit 4 further outputs the input block data of the current reduced frame SF0 as data of the position of the processing target block F1Ba as the test interpolation data TDB1 to TDBM, and the input second reduced delay The block data of the frame SF2 that is used as the data at the position of the processing target block F1Ba is output as test interpolation data TDC1 to TDCM.
For the sake of distinction, the test interpolation data TDA1 to TDAM are referred to as the first group of first to Mth test interpolation data, and the test interpolation data TDB1 to TDBM are referred to as the second group of first to Mth test interpolation data. The test interpolation data TDC1 to TDCM are referred to as a third group of first to Mth test interpolation data.

  The interpolation data evaluation unit 5 evaluates a plurality of test interpolation data with reference to the block data of the first reduced delay frame SF1, and outputs the evaluation data ED to the motion vector determination unit 6. In this evaluation, a correlation between the test interpolation data and the block data of the first reduced delay frame SF1 is obtained, and a higher evaluation is given as the correlation is stronger.

  The motion vector determination unit 6 generates and outputs a first reduced motion vector SMV1 and a second reduced motion vector SMV2 based on the evaluation data ED.

  The motion vector expansion unit 7 expands the values of the first reduced motion vector SMV1 and the second reduced motion vector SMV2 to generate and output the first motion vector MV1 and the second motion vector MV2. Here, the enlargement rate is equal to the reciprocal of the reduction rate α when the reduced frames SF0, SF1, and SF2 are generated, and is expressed by 1 / α.

  Next, specific examples of the test interpolation unit 4, the interpolation data evaluation unit 5, and the motion vector determination unit of the motion vector detection unit 12 will be described in more detail with reference to FIGS.

  The test interpolation unit 4 includes a plurality of, that is, first to Mth test interpolation data generation units 4-1 to 4-M, and the interpolation data evaluation unit 5 includes the first to Mth correlation values of the first group. Calculation units 5A-1 to 5A-M, first to Mth correlation value calculation units 5B-1 to 5B-M in the second group, and first to Mth correlation value calculation units 5C- in the third group. 1-5C-M.

The test interpolation data generation units 4-1 to 4-M generate the data F0B1 to F0BM of the block of the current reduced frame SF0 and the data F2B1 to F2BM of the block of the second reduced delay frame SF2 that make a pair with them respectively for each pixel. The data consisting of the average value obtained by averaging is calculated as the first group of test interpolation data TDA1 to TDAM. In the test interpolation data generation units 4-1 to 4 -M, data used for obtaining an average is data of blocks at different positions.
Test Interpolation Unit 4 Also, as described above, block data F0B1 to F0BM of the current reduced frame SF0 are used as the second group of test interpolation data TDB1 to TDBM, and block data F2B1 to F2BM of the second reduced delay frame SF2 are used. Output as the third group of test interpolation data TDC1 to TDCM. In FIG. 1, a set of test interpolation data TDA1 to TDCM is represented by a symbol TD.

This will be described in more detail below.
The first block data F0B1 of the current reduced frame SF0 and the first block data F2B1 of the second reduced delay frame SF2 are input to the test interpolation data generation unit 4-1.

  The test interpolation data generation unit 4-1 calculates the average value for each pixel of the first block data F0B1 of the current reduced frame SF0 and the first block data F2B1 of the second reduced delay frame SF2 in the first group. It outputs to correlation value calculation part 5A-1 as test interpolation data TDA1. Here, the average value for each pixel is the pixel value of each pixel in the block in the current reduced frame SF0 and the pixel at the corresponding position in the block in the second reduced delay frame SF2 (for example, the reference position of each block, For example, it means an average value of pixel values represented by the same coordinate value with the upper left corner as the origin.

  Similarly, the second block data F0B2 of the current reduced frame SF0 and the second block data F2B2 of the second reduced delay frame SF2 are input to the test interpolation data generation unit 4-2. The test interpolation data generation unit 4-2 calculates the average value for each pixel of the second block data F0B2 of the current reduced frame SF0 and the second block data F2B2 of the second reduced delay frame SF2 in the second group of the first group. It outputs to correlation value calculation part 5A-2 (not shown) as test interpolation data TDA2.

  Similarly, the test interpolation data generation units 4-3 (not shown) to 4-M also apply the third block data F0B3 to Mth block data F0BM of the current reduced frame SF0 and the third block of the second reduced delay frame SF2. Based on the data F2B3 to Mth block data F2BM, the first group of third test interpolation data TDA3 to Mth test interpolation data TDAM is generated, and correlation value calculation units 5A-3 (not shown) to 5A-M are generated. Output to.

  Generally speaking, the test interpolation data generation unit 4-m converts the mth block data F0Bm (m = 1 to M) of the current reduced frame SF0 and the mth block data F2Bm of the second reduced delay frame SF2. Based on this, for example, an average for each pixel is obtained, and the m-th test interpolation data TDm of the first group is generated and output to the correlation value calculation unit 5A-m.

Test interpolation unit 4 Furthermore, block data F0B1 to F0BM of the current reduced frame SF0 are output to the correlation value calculation units 5B-1 to 5B-M as test interpolation data TDB1 to TDBM, and the second reduced delay frame SF2 is output. The block data F2B1 to F2BM are output to the correlation value calculation units 5B-1 to 5C-M as test interpolation data TDC1 to TDCM.
Generally speaking, the test interpolation unit 4 outputs the mth block data F0Bm of the current reduced frame SF0 to the correlation value calculation unit 5B-m as the mth test interpolation data TDBm of the second group. The data F2Bm of the mth block of the second reduced delay frame SF2 is output to the correlation value calculation unit 5C-m as the mth test interpolation data TDCm of the third group.

  Data of the processing target block F1Ba of the first reduced delay frame SF1 is input to the correlation value calculation units 5A-1 to 5C-M in the interpolation data evaluation unit 5.

  Correlation value calculation units 5A-1 to 5C-M are respectively correlation values of test interpolation data TDA1 to TDCM output from test interpolation unit 4 and data of processing target block F1Ba of first reduced delay frame SF1, for example, absolute difference The sum of values is calculated and output as evaluation data EDA1 to EDCM.

  The correlation value calculation unit 5A-1 calculates the sum of absolute differences between the data of each pixel of the first group of first test interpolation data TDA1 and the data of each pixel of the processing target block F1Ba of the first reduced delay frame SF1. Calculated and output to the motion vector determination unit 6 as evaluation data EDA1. The sum of absolute differences is expressed by the following equation (1).

  Here, BK1 and BK2 are the data of each pixel in the block, and BK1 is the data of each pixel constituting the test interpolation data TDA1, and BK2 is the data of each pixel in the processing target block F1Ba. This is the sum of absolute differences between the data of each pixel of the first group of first test interpolation data TDA1 and the data of each pixel of the processing target block F1Ba of the first delay frame F1. The difference absolute value sum SAD represents the degree of correlation, and is output from the correlation value calculation unit 5A-1 as evaluation data EDA1.

  The difference absolute value sum given by equation (1) means that the smaller the value, the stronger the correlation. When the difference absolute value sum SAD is used as evaluation data, the smaller the value, the higher the evaluation. Represent.

  Similarly, the correlation value calculation units 5A-2 to 5C-M also process blocks of the first group of second test interpolation data TDA2 to the third group of M test interpolation data TDCM and the first reduced delay frame SF1. The sum of absolute differences of the F1Ba data is calculated and output to the motion vector determination unit 6 as evaluation data EDA2 to EDCM.

  Of the evaluation data EDA1 to EDCM, the data corresponding to the first group of test interpolation data TDA1 to TDAM is referred to as the first group of evaluation data EDA1 to EDAM, and the data corresponding to the second group of test interpolation data TDB1 to TDAM. The second group of evaluation data EDB1 to EDBM is referred to as the third group of test interpolation data TDC1 to TDCM, and the third group of evaluation data EDC1 to EDCM is referred to as the third group of evaluation data EDC1 to EDCM.

  The motion vector determination unit 6 finds the one having the smallest value among the evaluation data EDA1 to EDCM (indicating the highest evaluation).

  When the evaluation data EDA1 to EDCM having the smallest value is any one of the first group of evaluation data EDA1 to EDAM, the motion vector determination unit 6 performs the processing target block F1Ba of the first reduced delay frame SF1. The relative position (positional difference) of the block in the second reduced delay frame SF2 corresponding to the evaluation data of the minimum value with respect to the position of is output as the first reduced motion vector SMV1, and the first reduced delay frame The relative position (positional difference) of the block in the current reduced frame SF0 corresponding to the evaluation data of the minimum value with respect to the position of the processing target block F1Ba of SF1 is output as the second reduced motion vector SMV2. Note that the block in the second reduced delay frame SF2 and the block in the current reduced frame SF0 corresponding to the same evaluation data are in a point-symmetrical position with respect to the processing target block F1Ba, so that SMV2 = −SMV1. There is.

  When the evaluation data EDA1 to EDCM having the smallest value is any one of the second group of evaluation data EDB1 to EDBM, the motion vector determination unit 6 performs the processing target block F1Ba of the first reduced delay frame SF1. The relative position (positional difference) of the block in the current reduced frame SF0 corresponding to the evaluation data of the minimum value is output as the second reduced motion vector SMV2, and instead of the first reduced motion vector SMV1 A signal (data or flag) indicating that there is no block corresponding to (no block with sufficiently small sum of absolute differences), that is, there is no valid data is output.

When the evaluation data EDA1 to EDCM having the smallest value is any one of the third group of evaluation data EDB1 to EDCM, the motion vector determination unit 6 performs the processing target block F1Ba of the first reduced delay frame SF1. The relative position (positional difference) of the block in the second reduced delay frame SF2 corresponding to the evaluation data of the minimum value with respect to the position of is output as the first reduced motion vector SMV1, and the second reduced motion vector Instead of SMV2, a signal indicating that there is no corresponding block (no block having a sufficiently small difference absolute value sum), that is, no valid data is output.

  The motion vector enlargement unit 7 enlarges the reduced motion vectors SMV1 and SMV2 by the following equations (2A) and (2B), and outputs the motion vectors MV1 and MV2. Here, the reduced motion vectors SMV1 and SMV2 are values in the reduced image (reduced frame reference, that is, image-based motion vector), and the motion vectors MV1 and MV2 are values in the input image (input frame reference, that is, input image-based motion). Vector). That is, the reduced motion vectors SMV1 and SMV2 are motion vectors from the second reduced delay frame SF2 to the first reduced delay frame SF1 and from the current reduced frame SF0 to the first reduced delay frame SF1, respectively, and the motion vectors MV1 and MV2 is a motion vector from the second delay frame F2 to the first delay frame F1, and from the current frame F0 to the first delay frame F1, respectively.

MV1 = SMV1 / α (2A)
MV2 = SMV2 / α (2B)

  However, when a signal indicating no corresponding block is output instead of the reduced motion vector SMV1, a signal indicating no corresponding block is output instead of the motion vector MV1, and no corresponding block is indicated instead of the reduced motion vector SMV2. When a signal is output, a signal indicating no corresponding block is output instead of the motion vector MV2.

FIG. 5 is a diagram for explaining the operation of the motion vector detection unit 12. Here, the case where M = 1 will be described.
A partial area of the first reduced delay frame SF1 is cut out as a processing target block F1Ba of the first reduced delay frame SF1.

  An area corresponding to the position shifted by the vector −V1 with respect to the processing target block F1Ba of the first reduced delay frame SF1 is set and cut out as the first block data F2B1 of the second reduced delay frame SF2, and shifted by the vector + V. The area corresponding to the selected position is set and cut out as the first block data F0B1 of the current reduced frame SF0.

The test interpolation data generation unit 4-1 generates the test interpolation data TDA1 by averaging the first block data F0B1 of the current reduced frame SF0 and the first block data F2B1 of the second reduced delay frame SF2 for each pixel. .
Further, the test interpolation unit 4 outputs the first block data F0B1 of the current reduced frame SF0 as test interpolation data TDB1, and the first block data F2B1 of the second reduced delay frame SF2 as test interpolation data TDC1.

  The correlation value calculation unit 5A-1 calculates the difference absolute value sum SAD from the test interpolation data TDA1 and the data of the processing target block F1Ba of the first reduced delay frame SF1 using the above equation (1), and the evaluation data EDA1 Output as.

Similarly, the correlation value calculator 5B-1 calculates the difference absolute value sum SAD from the test interpolation data TDB1 and the data of the processing target block F1Ba of the first reduced delay frame SF1 using the above equation (1), and evaluates it. Output as data EDB1.
Similarly, the correlation value calculation unit 5C-1 calculates the difference absolute value sum SAD from the test interpolation data TDC1 and the data of the processing target block F1Ba of the first reduced delay frame SF1 using the above equation (1) and evaluates it. Output as data EDC1.

  The motion vector determination unit 6 reduces the motion vector based on the relative position of the block or block pair that generated the minimum value of the evaluation data EDA1, EDB1, and EDC1 and the processing target block F1Ba in the first reduced delay frame SF1. Ask for.

  When the evaluation data EDA1 is the minimum value, the relative position of the block in the second reduced frame SF2 with respect to the processing target block F1Ba in the first reduced delay frame SF1 is expressed as the first reduced motion vector SMV1 (= V1). And the relative position of the block in the current reduced frame SF0 with respect to the processing target block F1Ba in the first reduced delay frame SF1 is output as the second reduced motion vector SMV2 (= −V1).

  When the evaluation data EDB1 is the minimum value, a signal indicating no corresponding block is output instead of the first reduced motion vector SMV1, and the signal in the current reduced frame SF0 for the processing target block F1Ba in the first reduced delay frame SF1 is output. Is output as the second reduced motion vector SMV2 (= −V1).

  When the evaluation data EDC1 is the minimum value, the relative position of the block in the second reduced delay frame SF2 with respect to the processing target block F1Ba in the first reduced delay frame SF1 is the first reduced motion vector SMV1 (= V1). And a signal indicating no corresponding block is output instead of the second reduced motion vector SMV2.

  Although the case of M = 1 has been described in FIG. 5, the configuration of the embodiment according to the present invention is not limited to M = 1. That is, two or more block pairs corresponding to candidate vectors may be set. For example, with respect to the processing target block F1Ba in the first reduced delay frame, all the blocks in the second reduced delay frame located within the search range corresponding to the predetermined motion amount and point symmetric positions thereof. Test interpolation data may be generated by calculating an average value for a block pair consisting of blocks in the current reduced frame.

The motion vector detection unit 12 uses the data in the first reduced delay frame SF1, which is actual data as described above, to make up a block consisting of a block in the second reduced delay frame SF2 and a block in the current reduced frame SF0. Since the motion vector is determined by evaluating the vector candidates corresponding to the pair, the motion vector from the second reduced delay frame SF2 and the current reduced frame SF0 to the first reduced delay frame SF1 is accurately calculated. Can do.
Furthermore, since only the data in the second reduced delay frame SF2 or only the data in the current reduced frame SF0 is used as test interpolation data corresponding to the vector candidate, the second reduced delay frame to the current reduced frame are changed. Even if an object is hidden or appears in between, a motion vector can be calculated with high accuracy.

  The interpolation data evaluation unit 5 calculates the evaluation data using the sum of absolute differences, but there are many other functions for obtaining the correlation, such as the sum of square errors, and these can be replaced. That is, the interpolation data evaluation unit 5 only needs to obtain the correlation between the test interpolation data and the data of the processing target block, and the index (correlation value) indicating the correlation is other than the sum of absolute differences. May be.

Further, the motion vector determination unit 6 assigns different weights between the test interpolation data TDA1 to TDCM, thereby determining that a vector candidate corresponding to any one of the test interpolation data is determined to be a desired motion vector. You may make it raise.
For example, the value obtained by multiplying the sum of absolute differences obtained by the correlation value calculation units 5B-1 to 5C-M by the coefficient β1 is output as the evaluation data EDB1 to EDCM, and the correlation value calculation units 5A-1 to 5A-M A value obtained by multiplying the obtained sum of absolute differences by a coefficient β2 smaller than the coefficient β1 is output as evaluation data EDA1 to EDAM, and the motion vector determination unit 6 determines that the value of these evaluation data EDA1 to EDCM is You may try to find the smallest one. In this way, it is possible to increase the possibility that the block pair corresponding to the test interpolation data TDA1 to TDAM is determined to correspond to the motion vector obtained. In other words, a vector corresponding to a block pair corresponding to the test interpolation data TDA1 to TDAM can be preferentially selected as a motion vector.

  Conversely, the sum of absolute differences obtained by the correlation value calculation units 5B-1 to 5C-M is multiplied by the coefficient β1 and output as evaluation data EDB1 to EDCM, and the correlation value calculation units 5A-1 to 5A-M. Is obtained by multiplying the sum of absolute differences obtained by the above by the coefficient β2 larger than the coefficient β1 as the evaluation data EDA1 to EDAM, and the motion vector determination unit 6 determines the value of the evaluation data EDA1 to EDCM. You may try to find the smallest one. In this way, it is possible to increase the possibility that the block corresponding to the test interpolation data TDB1 to TDCM is determined to correspond to the desired motion vector. In other words, a vector corresponding to a block corresponding to the test interpolation data TDB1 to TDCM can be preferentially selected as a motion vector.

In addition to such weighting, weighting due to other factors may be added.
For example, among the blocks in the second reduced delay frame SF2 and the current reduced frame SF0, a block or a block pair close to the position of the processing target block F1Ba in the first reduced delay frame SF1 (the absolute value of the corresponding vector is small) It is also possible to increase the possibility that a closer block or block pair is determined to correspond to the desired motion vector by multiplying the sum of absolute differences obtained for the block) by a smaller coefficient.
In short, the evaluation data only needs to be determined based on the sum of absolute differences or other correlation values.

  Next, operations of the motion vector conversion unit 13 and the interpolation frame generation unit 14 will be described in more detail with reference to FIGS.

  The motion vector conversion unit 13 obtains the first motion vector MV1 from the second delay frame F2 to the first delay frame F1 and the second motion vector MV2 from the current frame F0 to the first delay frame F1. The third motion vector MV3 from one delay frame F1 to the interpolation frame IF and the fourth motion vector MV4 from the current frame F0 to the interpolation frame IF are converted.

  As shown in FIGS. 6 to 8, when the time interval of the input frame is t1, and the time interval from the first delay frame F1 to the interpolation frame IF is t2, the motion is expressed by the following equations (3A) and (3B). Vectors MV3 and MV4 are calculated. For example, when converting an input image signal of 60 Hz to an image signal of 120 Hz, t1 is 1/60 seconds and t2 is 1/120 seconds.

MV3 = MV1 × t2 / t1 (3A)
MV4 = MV2 × (t1−t2) / t1 (3B)

  However, when a signal indicating no corresponding block is output instead of the motion vector MV1, a signal indicating no corresponding block is output instead of the motion vector MV3, and a signal indicating no corresponding block is output instead of the motion vector MV2. If it is output, a signal indicating no corresponding block is output instead of the motion vector MV4.

  When the third and fourth motion vectors MV3 and MV4 are obtained in this way, the data of the first delay frame F1 before reduction at the position of the vector -MV3 from the interpolation frame IF as shown in FIGS. (Data in block F1Bb corresponding to processing target block F1Ba) and / or data of current frame F0 before reduction (data of block having the same size as block F1Bb) at the position of vector-MV4 from interpolation frame IF The data of the interpolation frame IF (the data of the block IFBb having the same size as the block F1Bb) is obtained based on the above.

For example, as shown in FIG. 6, when both the third and fourth motion vectors MV3 and MV4 are valid data (in the case of “no corresponding block”), the first delay frame described above is used. The average of the data in F1 and the data in the current frame F0 is calculated as the data of the interpolation frame IF. The interpolation frame IF data composed of the generated data is arranged and output between the data of the first delay frame F1 and the data of the current frame F0.
At this time, block data of the first delay frame F1 used for interpolation (data of the block F1Bb corresponding to the processing target block F1Ba of the first reduced delay frame SF1) and block data of the current frame F0 (current reduced frame SF0) And a block corresponding to the block F0B1) are symmetrical with respect to the position of data obtained by interpolation in the interpolation frame IF.

As shown in FIG. 7 or FIG. 8, when a “no corresponding block” signal is output instead of one of the third and fourth motion vectors MV3 and MV4, the other motion vector (third and The data in the first delay frame F1 or the data in the current frame F0 corresponding to the fourth motion vector MV3 and MV4) is the data of the interpolation frame IF (block having the same size as the block F1Bb). (IFBb data). FIG. 7 shows a case where a “no corresponding block” signal is output instead of the fourth motion vector MV4, and FIG. 8 shows that a “no corresponding block” signal is output instead of the third motion vector MV3. Indicates the case of output. The process of outputting data in one frame as it is as interpolation frame data in this way is called “zero-order interpolation”.
The interpolation frame generation unit 14 arranges and outputs the data of the interpolation frame IF composed of the data obtained in this way between the data of the first delay frame F1 and the data of the current frame F0.

  FIG. 9 is a diagram for explaining a correspondence relationship between the data of the current frame F0, the data of the first delay frame F1, and the data of the second frame F2 input to the motion vector detection unit 12. The operation of this embodiment when a signal representing a video as shown in FIG. 9 is input will be described. In the image shown in FIG. 9, a circle BC indicated by cross hatching moves from the upper left to the lower right of the screen with time, and the English character (no movement) behind it is hidden or visible.

FIG. 10 is a diagram showing image data representing a reduced image obtained by reducing the input image shown in FIG. However, in FIG. 10, the image is drawn in the same size as FIG. The elements of the image of FIG. 9 (characters indicating the circles BC and A to J appear in FIG. 10 as they are, and therefore the English characters A to J are hidden or visible behind the circle BC.
FIGS. 11 to 17 are diagrams for explaining the operations of the test interpolation unit 4 and the interpolation data evaluation unit 5. FIG. 18 is a diagram for explaining the operation of the motion vector enlargement unit 7.

  The operation of the motion vector detector 12 when the data of the second reduced delay frame SF2, the data of the first reduced delay frame SF1, and the data of the current reduced frame SF0 as shown in FIG. 10 are input will be described. Here, M = 2 is assumed.

The test interpolation unit 4 generates test interpolation data for each motion vector candidate.
For example, as shown in FIG. 11, a partial area of the first reduced delay frame SF1 is a processing target block F1Ba of the first reduced delay frame SF1, and the processing target block F1Ba of the first reduced delay frame SF1 is −V1. The block of the second reduced delay frame SF2 at the shifted position is set as the first block F2B1, and the block of the current reduced frame SF0 at the position shifted by V1 from the processing target block F1Ba of the first reduced delay frame SF1 is set as the block F0B1. .
Further, the block of the second reduced delay frame SF2 at the position shifted by −V2 from the processing target block F1Ba of the first reduced delay frame SF1 is referred to as a block F2B2, and is shifted by V2 from the processing target block F1Ba of the first reduced delay frame SF1. The block of the current reduced frame SF0 at the position is designated as block F0B2.

As shown in FIG. 12, an average for each pixel obtained from the data of the block F2B1 and the data of the block F0B1 is generated as the test interpolation data TDA1.
Similarly, as shown in FIG. 13, an average for each pixel is generated as test interpolation data TDA2 from the data in block F2B2 and the data in block F0B2.
Further, as shown in FIGS. 14 and 15, the data of the blocks F0B1 and F0B2 are output as test interpolation data TDB1 and TDB2, respectively.
Similarly, as shown in FIGS. 16 and 17, the data of the blocks F2B1 and F2B2 are output as test interpolation data TDC1 and TDC2, respectively.

  The interpolation data evaluation unit 5 calculates the sum of absolute differences between each of the test interpolation data TDA1 to TD6 shown in FIGS. 12 to 17 and the data of the processing target block F1Ba, and outputs evaluation data EDA1 to ED6. In the specific examples of FIGS. 10 to 17, the area representing the character “F” on the first reduced delay frame SF1 is the processing target block F1Ba, and this character “F” is the second reduced delay frame SF2. Although it is visible in the first reduced delay frame SF1, it is hidden behind the circle BC indicated by hatching in the current reduced frame F0. Therefore, from the region representing the character “F” in the second reduced delay frame SF2. The sum of absolute differences between the test interpolation data TD5 obtained from the block F2B1 and the processing target block F1Ba, that is, the evaluation data ED5 is minimized.

  The motion vector determination unit 6 outputs the reduced motion vector SMV1 (= V1) corresponding to the smallest evaluation data ED5 among the evaluation data EDA1 to ED6, and outputs a signal indicating that there is no corresponding block instead of the reduced motion vector SMV2. To do.

The motion vector enlargement unit 7 uses the above equations (2A) and (2B) for the reduced motion vectors SMV1 and SMV2, and the motion vector MV1 from the second delay frame F2 to the first delay frame F1 A motion vector MV2 from the current frame F0 to the first delay frame F1 is output (FIG. 18).
As described above, when a signal indicating no corresponding block is output instead of one of the reduced motion vectors, the motion vector (MV1 or MV2) corresponding to the reduced motion vector is also output. Instead, a signal indicating “no corresponding block” is output. Therefore, in the examples of FIGS. 10 to 17, the motion vector MV1 (= V1 / α) is output, and a signal indicating no corresponding block is used instead of MV2. Is output.

  By setting a block without any gap (no gap) and calculating a reduced motion vector for the first reduced delay frame SF1 obtained by reducing the first delayed frame F1, the first and second portions of the first reduced delay frame SF1 are all calculated. Generating two reduced motion vectors SMV1 and / or SMV2 and expanding the first and second reduced motion vectors SMV1, SMV2 to obtain first and second motion vectors MV1 and / or MV2, For the pixel in the interpolation frame IF at the position corresponding to the pixel in the first delay frame F1 corresponding to the pixel in the first reduced delay frame SF1, for example, at the same position, the first and second It is also possible to obtain the third and fourth motion vectors MV3 and MV4 by performing conversion using the motion vectors MV1 and MV2. .

  Instead, by performing the above processing on a block having a predetermined size centered on each pixel in the first reduced delay frame SF1 obtained by reducing the first delay frame F1, the first delay frame F1 is processed. The reduced motion vector SMV1 and the second reduced motion vector SMV2 are generated, and the first and second reduced motion vectors SMV1 and SMV2 are expanded to obtain the first and second motion vectors MV1 and / or MV2. The first and second pixels corresponding to the pixels in the first delay frame F1 corresponding to the pixels in the first reduced delay frame SF1, for example, the pixels in the interpolation frame IF at the same position The third and fourth motion vectors MV3 and MV4 may be obtained by performing conversion using the motion vectors MV1 and MV2.

  FIGS. 19A and 19B are diagrams for explaining a specific example of the operation of the motion vector conversion unit 13. FIG. 19A shows the input of the motion vector conversion unit 13, and FIG. 19B shows the output of the motion vector conversion unit 13.

  The motion vector conversion unit 13 performs the first motion vector MV1 from the second delay frame F2 to the first delay frame F1 and the current frame F0 to the first delay frame F1 shown in FIG. The third motion vector MV3 from the first delay frame F1 to the interpolated frame IF shown in FIG. 19 (b) using the above equations (3A) and (3B) for the motion vector MV2 of The current frame F0 is converted into a fourth motion vector MV4 from the interpolation frame IF. In the specific examples shown in FIGS. 19A and 19B, MV1 outputs a signal indicating no corresponding block instead of V1 / α and MV2, t1 being 1/60 second, and t2 being 1 / t of t1. Is equal to 2 and 1/120 second, and from the above equations (3A) and (3B), MV3 outputs a signal indicating no corresponding block in place of V1 / 2α and MV4.

FIG. 20 is a diagram for explaining a specific example of the operation of the interpolation frame generation unit 14. In FIG. 20, as in FIGS. 9 to 17, the area representing the character “F” on the first delay frame F1 is a block (non-reduced block) F1Bb corresponding to the processing target block F1Ba. “F” is visible in the second delay frame F2 and the first delay frame F1, but is hidden behind the circle BC indicated by cross-hatching in the current frame F0, and therefore in the second delay frame F2. This shows a case where the sum of absolute differences between the test interpolation data TD5 obtained from the block F2B1 including the region representing the character “F” and the processing target block F1Ba, that is, the evaluation data ED5 is minimized. In this case, as shown in FIG. 20, the data of the first delay frame F1 at the position moved by -MV3 from the interpolation frame IF is output as the data of the interpolation frame IF.
The interpolation frame IF generated by performing such processing for all the blocks in the interpolation frame IF is interpolated between the first delay frame F1 and the current frame F0 and output.

  As described above, in the present embodiment, motion vectors are detected with high accuracy, and interpolation frames can be interpolated without image distortion. Furthermore, even when an object is hidden or appears before and after the interpolated frame, the interpolated frame can be interpolated without image distortion.

Here, the case where the image reduction process is performed in the current frame dither reduction unit 21, the first delay frame dither reduction unit 22, and the second delay frame dither reduction unit 23 is compared with the case where the image reduction process is not performed. When image reduction processing by the dither reduction units 21, 22, and 23 is not performed, the input image data is processed as it is in the motion vector detection unit 12, and when image reduction processing by the dither reduction units 21, 22, and 23 is performed, a motion vector detection unit In 12, the reduced image data is processed. At this time, when searching for a motion vector, if a range of vectors having the same size is searched, it is possible to search a wider range by using a reduced image, and therefore, a large motion can be detected.
In the case of using a reduced image, the number of pixels included in each block is smaller than the number of pixels included in each block of the non-reduced image, and is used for calculation in the calculation of the sum of absolute differences for each block. Since the number of pixels to be obtained is small, the amount of calculation is reduced, the response is improved, and the circuit scale can be reduced.

Hereinafter, an effect of dithering reduction of an image will be described.
FIGS. 21A and 21B and FIGS. 22A and 22B are diagrams for explaining the effect of dither reduction. FIG. 21A and FIG. 22A show examples of pixel value change patterns in the original image. FIGS. 21B and 22B show the reductions obtained by reducing the images of FIGS. 21A and 22A in the one-dimensional direction by a normal method (without using dither reduction). The pattern of a change in pixel value in an image is shown. Assume that a region similar to the images in FIGS. 21A and 21B is searched in the images in FIGS. 22A and 22B. In the image before reduction in FIG. 22A (in the range shown in the figure), the area PBa is easily similar to the area surrounded by the dotted line PAa in the image before reduction in FIG. Detected. On the other hand, for the area QAa in the reduced image in FIG. 21B, it is determined that the area QBb is more similar to the area QBa in the reduced image in FIG. Occurs. This is because in the reduced image, when the phase in which the pixels are arranged is made constant, the change pattern of the pixel value may greatly change depending on the phase. On the other hand, in dither reduction, the phase dependency of the pixel value change pattern in the reduced image can be averaged by shifting the phase at random, and as a result, matching errors as described above can be reduced. .

Although the present invention has been described above as an image processing apparatus and an image display method, the image processing method and the image display method implemented by the above apparatus also form part of the present invention.
Further, although the case where the present invention is realized by hardware has been described, part or all of the configuration shown in FIG. 1 can be realized by software. Processing in that case will be described with reference to FIGS.

  FIG. 23 is composed of a computer system for realizing the processing by the frame memory 11, the motion vector detection unit 12, the motion vector conversion unit 13 and the interpolation frame generation unit 14 in the configuration shown in FIG. 1 by software. 1 shows an image processing apparatus. The illustrated image processing apparatus includes a processor 101, a program memory 102, a data memory 103, an input interface 104, an output interface 105, and a bus 106 for connecting them.

The processor 101 operates according to a program stored in the program memory 102 and performs processing on an image input via the input interface 104. Various data are stored in the data memory 103 in the course of operation. An image generated as a result of the processing is supplied to the image display unit 15 shown in FIG. 1 via the interface 105 and used for display by the image display unit 15.
Hereinafter, processing performed by the processor 101 will be described with reference to FIG.

  FIG. 24 shows a flow of an image processing method implemented by the image processing apparatus of FIG.

  First, in the motion vector detection step ST1, the second reference is made with reference to the image data F0, the image data F1 delayed by one frame with respect to the image data F0, and the image data F2 delayed by two frames with respect to the image data F0. A first motion vector MV1 from the delay frame F2 to the first delay frame F1 and a second motion vector V2 from the current frame F0 to the first delay frame F1 are generated. This operation is equivalent to the motion vector detection unit 12 of FIG.

  In the motion vector conversion step ST2, the first and second motion vectors MV1 and MV2 are transferred from the first delay frame F1 to the interpolation frame IF (inserted between the current frame F0 and the first delay frame F1). The third motion vector MV3 and the fourth motion vector MV4 from the current frame F0 to the interpolation frame IF are converted. This operation is equivalent to the motion vector conversion unit 13 in FIG.

  In the interpolation frame generation step ST3, data of the interpolation frame IF is generated from the data of the first delay frame F1, the data of the current frame F0, the third motion vector MV3, and the fourth motion vector MV4. The above processing is repeated for all the blocks of the interpolation frame IF, and image data DO is generated by inserting the data of the generated interpolation frame IF between the data of the current frame F0 and the data of the first delay frame F1. To do. This operation is equivalent to the interpolation frame generation unit 14 of FIG.

  FIG. 24 realizes a part of the image display apparatus according to the present embodiment as software, and the effect is the same as in the case of FIG.

  2 dither reduction unit, 3 block cutout unit, 4 test interpolation unit, 5 interpolation data evaluation unit, 6 motion vector determination unit, 7 motion vector enlargement unit, 10 image processing device, 11 frame memory, 12 motion vector detection unit, 13 motion Vector conversion unit, 14 interpolation frame generation unit, 15 image display unit, 21 current frame dither reduction unit, 22 first delay frame dither reduction unit, 23 second delay frame dither reduction unit, 31 current frame block cutout unit, 32 A first delayed frame block cutout unit, 33 a second delayed frame block cutout unit.

Claims (16)

In an image processing apparatus for inserting an interpolation frame between a current frame of an image and a first delay frame which is a frame one frame before the current frame,
The first movement from the second delay frame to the first delay frame with reference to the data of the first delay frame and the data of the second delay frame which is a frame two frames before the current frame A motion vector detector that calculates a vector and calculates a second motion vector from the current frame to the first delay frame by referring to the data of the current frame and the data of the first delay frame;
The first motion vector is converted into a third motion vector from the first delay frame to the interpolation frame, and the second motion vector is converted into a fourth motion vector from the current frame to the interpolation frame. A motion vector conversion unit to convert,
The interpolation frame data is generated from the third motion vector, the fourth motion vector, the first delay frame data, and the current frame data, and the interpolation frame data is converted into the current frame data. An interpolation frame generator for outputting image data inserted between the data and the data of the first delay frame,
The motion vector detection unit
The current frame is reduced by shifting the phase at random for each pixel, and the current reduced frame is output. The first delayed frame is reduced by shifting the phase at every pixel, and the first reduced delay frame is output. A dither reduction unit that outputs the second reduced delay frame by reducing the phase of the second delayed frame at random for each pixel, and
An image processing apparatus comprising: a motion vector expansion unit that expands and outputs a motion vector detected from the current reduced frame, the first reduced delay frame, and the second reduced delay frame. The motion vector detection unit
A block in the first reduced delay frame corresponding to the interpolation target block of the interpolation frame is set as a processing target block, and data of a block forming a part of the second reduced delay frame and the current reduced frame A first group of test interpolation data composed of an average value for each pixel of block data forming a part; a second group of test interpolation data composed of block data forming part of the current frame; A test interpolation unit that outputs a plurality of test interpolation data including a third group of test interpolation data composed of block data of the second reduced delay frame;
Interpolation data that evaluates the strength of correlation of each test interpolation data with respect to the data of the processing target block among the first to third groups of test interpolation data and outputs a plurality of evaluation data indicating the results of the evaluation An evaluation unit;
A motion vector corresponding to the test interpolation data that has generated the evaluation data indicating the strongest correlation among the plurality of evaluation data is output as at least one of the first reduced motion vector and the second reduced motion vector. A motion vector determination unit;
The motion vector expansion unit expands at least one of the first reduced motion vector and the second reduced motion vector and outputs it as at least one of the first and second motion vectors;
The first group of test interpolation data is obtained from a plurality of blocks of the second reduced delay frame and a plurality of blocks of the current reduced frame;
The second group of test interpolation data is obtained from a plurality of blocks of the current reduced frame;
The image processing apparatus according to claim 1, wherein the third group of test interpolation data is obtained from a plurality of blocks of the second reduced delay frame. The test interpolation unit includes:
A plurality of blocks each consisting of a plurality of pixels in each of the first reduced delay frames are sequentially selected as processing target blocks, and the processing target block is the center.
A block in the second reduced delay frame and a block in the current reduced frame, which are located symmetrically with respect to each other, are extracted, and data of the extracted block of the second reduced delay frame and the extracted current block are extracted. the average value of each pixel data of the block of the reduced small frame, determined as test interpolation data of the first group,
Outputting the data of the block in the extracted current reduced frame as the second group of test interpolation data;
The image processing apparatus according to claim 2, wherein the extracted data of the block in the second reduced delay frame is output as the third group of test interpolation data. The test interpolation unit has a plurality of test interpolation data generation units,
Each of the plurality of test interpolation data generation units averages the data of different position blocks in the current reduced frame and the data of blocks in different positions in the second reduced delay frame for each pixel. The image processing apparatus according to claim 3, wherein the obtained data is calculated as the first group of test interpolation data. The interpolation data evaluation unit
Each of the plurality of test interpolation data output from the test interpolation unit,
Calculating a correlation value indicating a correlation with the data of the processing target block;
The image processing apparatus according to any one of claims 2 to 4, wherein the plurality of evaluation data are generated based on a plurality of correlation values respectively obtained for the plurality of test interpolation data. The interpolation data evaluation unit
The difference absolute value sum that is a sum of absolute values of differences between the plurality of test interpolation data output from the test interpolation unit and data of the processing target block is calculated as the correlation value. Item 6. The image processing apparatus according to Item 5 . The interpolation data evaluation unit
The image processing apparatus according to claim 6, further comprising a plurality of correlation value calculation units that calculate a sum of absolute differences between the data of the processing target block and the plurality of test interpolation data. An image processing apparatus according to any one of claims 1 to 7,
An image display device comprising: an image display unit that displays an image based on the image data output from the interpolation frame generation unit. In an image processing method of inserting an interpolation frame between a current frame of an image and a first delay frame that is a frame one frame before the current frame,
The first movement from the second delay frame to the first delay frame with reference to the data of the first delay frame and the data of the second delay frame which is a frame two frames before the current frame A motion vector detecting step of calculating a vector and calculating a second motion vector from the current frame to the first delay frame with reference to the data of the current frame and the data of the first delay frame;
Converting the first motion vector into a third motion vector from the first delay frame to the interpolation frame, and converting the second motion vector into a fourth motion vector from the current frame to the interpolation frame; A motion vector conversion step to convert,
The interpolation frame data is generated from the third motion vector, the fourth motion vector, the first delay frame data, and the current frame data, and the interpolation frame data is converted into the current frame data. An interpolation frame generating step of outputting image data inserted between the data and the data of the first delay frame,
The motion vector detection step includes
The current frame is reduced by shifting the phase at random for each pixel, and the current reduced frame is output. The first delayed frame is reduced by shifting the phase at every pixel, and the first reduced delay frame is output. A dither reduction step of reducing the second delay frame by shifting the phase at random for each pixel and outputting a second reduced delay frame;
An image processing method comprising: a motion vector expansion step of expanding and outputting a motion vector detected from the current reduced frame, the first reduced delay frame, and the second reduced delay frame. The motion vector detection step includes
A block in the first reduced delay frame corresponding to the interpolation target block of the interpolation frame is set as a processing target block, and data of a block forming a part of the second reduced delay frame and the current reduced frame A first group of test interpolation data composed of an average value for each pixel of block data forming a part; a second group of test interpolation data composed of block data forming part of the current frame; A test interpolation step of outputting a plurality of test interpolation data including a third group of test interpolation data composed of block data of the second reduced delay frame;
Interpolation data that evaluates the strength of correlation of each test interpolation data with respect to the data of the processing target block among the first to third groups of test interpolation data and outputs a plurality of evaluation data indicating the results of the evaluation An evaluation step;
A motion vector corresponding to the test interpolation data that has generated the evaluation data indicating the strongest correlation among the plurality of evaluation data is output as at least one of the first reduced motion vector and the second reduced motion vector. A motion vector determination step,
The motion vector expansion step expands at least one of the first reduced motion vector and the second reduced motion vector and outputs it as at least one of the first and second motion vectors;
The first group of test interpolation data is obtained from a plurality of blocks of the second reduced delay frame and a plurality of blocks of the current reduced frame;
The second group of test interpolation data is obtained from a plurality of blocks of the current reduced frame;
10. The image processing method according to claim 9, wherein the third group of test interpolation data is obtained from a plurality of blocks of the second reduced delay frame. The test interpolation step includes:
A plurality of blocks each consisting of a plurality of pixels in each of the first reduced delay frames are sequentially selected as processing target blocks, and the processing target block is the center.
A block in the second reduced delay frame and a block in the current reduced frame, which are located symmetrically with respect to each other, are extracted, and data of the extracted block of the second reduced delay frame and the extracted current block are extracted. the average value of each pixel data of the block of the reduced small frame, determined as test interpolation data of the first group,
Outputting the data of the block in the extracted current reduced frame as the second group of test interpolation data;
11. The image processing method according to claim 10, wherein data of the block in the extracted second reduced delay frame is output as the third group of test interpolation data. The test interpolation step includes a plurality of test interpolation data generation steps,
In the plurality of test interpolation data generation steps, data of blocks in different positions in the current reduced frame and data of blocks in different positions in the second reduced delay frame are averaged for each pixel. The image processing method according to claim 11, wherein the obtained data is calculated as the first group of test interpolation data. The interpolation data evaluation step includes
Each of the plurality of test interpolation data output from the test interpolation step;
Calculating a correlation value indicating a correlation with the data of the processing target block;
The image processing method according to any one of claims 10 to 12, wherein the plurality of evaluation data are generated based on a plurality of correlation values respectively obtained for the plurality of test interpolation data. The interpolation data evaluation step includes
The difference absolute value sum that is a sum of absolute values of differences between the plurality of test interpolation data output from the test interpolation step and the data of the processing target block is calculated as the correlation value. Item 14. The image processing method according to Item 13 . The interpolation data evaluation step includes
The image processing method according to claim 14 , further comprising a plurality of correlation value calculation steps for calculating a sum of absolute differences between the data of the processing target block and the plurality of test interpolation data. An image processing method according to any one of claims 9 to 15,
An image display method comprising: an image display step for displaying an image based on the image data output from the interpolation frame generation step.

Images