JP2012257148A - Motion vector detection apparatus, encoder, decoder and program of them - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce an information amount of a motion vector and to obtain highly accurate motion vector information.SOLUTION: A motion vector detection apparatus includes: a block division section 10 for generating a block image obtained by dividing an original image; a spatial frequency analysis section 20 for generating a spatial frequency spectrum of the block image; a motion vector detection section 30 for detecting the motion vector of each of the block images; a spatial direction high frequency region power calculation section 41 for calculating a rate of spatial high frequency power from the spatial frequency spectrum; an average luminance difference calculation section 42 for calculating a differential value of average luminance among the plurality of adjacent block images; a block size correction section 43 for integrating the block images in a block set image and setting it to be an integrated block image when the rate of the high frequency power is a prescribed threshold or below and the differential value of the average luminance is a prescribed threshold or below; and a motion vector correction section 44 for outputting the average value of the motion vector of each of the block images as the motion vector of the integrated block image on the integrated block image.

Description

  The present invention relates to a motion vector detection device that detects a motion vector between a base frame and a reference frame of an image, an encoding device using the motion vector detection device, and a decoding that decodes encoded data transmitted from the encoding device. The present invention relates to a device and these programs.

  2. Description of the Related Art In recent years, high definition of imaging devices and display devices has progressed, and a high resolution technology for moving images called super-resolution has been studied (see, for example, Patent Document 1). Ultra-high definition video such as Super Hi-Vision (SHV) called 8K system, or high-definition video like Digital Cinema called 4K system has a resolution that is 4 to 16 times that of conventional high-definition (HDTV) video. It has come to have.

  Therefore, when super-resolution is performed using two images, after pre-processing such as scene change detection and pixel area mask processing that is not suitable for pixel alignment, super-resolution is performed by pixel reconstruction processing. A technique for creating an image is known (see, for example, Patent Document 2).

  Also, when super-resolution is performed using two images, the average luminance in the image of the source block and destination block is calculated, and the luminance change by adding an offset to the luminance of the destination block A technique is known in which after removal is performed, the similarity between the movement source block and the movement destination block is compared, and a super-resolution image is generated by pixel reconstruction processing (see, for example, Patent Document 3).

  The video format of the SHV screen is horizontal 7680 pixels, vertical 4320 lines, time 60 frames / second. Compared with the video format of the HDTV screen, the horizontal and vertical sampling frequencies are relative to the time sampling frequency. Has increased.

  Therefore, the moving area of the SHV screen shows a larger amount of movement (number of pixels indicating movement in units of frames: pixels / frame) than the HDTV screen when compared with moving images captured at the same angle of view. In the region, it is assumed that the correlation between frames is low and the power in the high frequency region in the time direction is high, the blur amount in the moving region is large, and the power in the high frequency region in the spatial direction is low. . Processing based on these assumptions is effective in estimating the amount of motion between a plurality of frames in encoding processing and super-resolution processing.

  Therefore, the resolution value is determined by detecting the power ratio in the high frequency region in the time direction and / or the power ratio in the low frequency region in the spatial direction, and the block size and motion search range corresponding to the determined resolution value are determined. Hierarchy in which motion vector detection is started from a size hierarchy, and gradually moves to motion vector detection in a hierarchy having a block size smaller than the block size and a size of the motion search range smaller than the size of the motion search range. A technique for detecting a motion vector of a type is known (see, for example, Patent Document 3).

JP 2009-105490 A JP 2010-134582 A JP 2011-082700 A

  Using the technique described in Patent Document 3, high-precision motion vector detection is performed by estimating an approximate amount of motion by performing spatio-temporal frequency analysis of an image and determining the number of layers for hierarchical motion detection. be able to. However, there are various objects in the image, and in particular, in a wide-field and high-resolution video such as SHV, the number of objects included in the image is very large. In order to perform an image, finer control of the motion detection block size, the motion search range, and the like is required within the screen.

  The present invention has been made in view of the above-described problems. The image feature is determined for each block of the original image, and the blocks capturing the same object are integrated, thereby optimizing the block size and the motion vector. Motion vector detection apparatus capable of reducing the amount of information and obtaining highly accurate motion vector information, an encoding apparatus using the motion vector detection apparatus, and a decoding apparatus for decoding encoded data transmitted from the encoding apparatus And to provide these programs.

  In order to solve the above problems, a motion vector detection device according to the present invention is a motion vector detection device that detects a motion vector between a base frame and a reference frame of an image, and converts the original image into blocks of a predetermined size. A block dividing unit for generating a divided block image, a spatial frequency analyzing unit for generating a spatial frequency spectrum by performing a spatial frequency analysis on the block image, and a motion vector between a reference frame and a reference frame for each block image A motion vector detection unit that detects a plurality of adjacent block images as a block set image, an average luminance difference calculation unit that calculates a difference value of average luminance between block images in the block set image, and the spatial frequency spectrum , One block image in the block set image is a high frequency region of spatial frequency. For the spatial direction high-frequency region power calculation unit that calculates the ratio of-and the block set image, the difference value of the average luminance is less than or equal to a predetermined threshold, and the power ratio of the high-frequency region is less than or equal to a predetermined threshold In some cases, a block size correction unit that integrates block images in the block set image into an integrated block image, and for the integrated block image, an average value of motion vectors of the respective block images is obtained from the integrated block image. For a block image that is output as a motion vector and is not integrated, a motion vector for each block image detected by the motion vector detection unit is output, and a motion indicating a block size of an image corresponding to the output motion vector A motion vector correction unit that outputs detection block information; and That.

  Furthermore, in the motion vector detection device according to the present invention, for the integrated block image, the motion vector correction unit calculates a difference value of motion vectors of each block image before integration in the integrated block image, and the difference value Is equal to or less than a predetermined threshold, an average value of motion vectors of the respective block images before integration is output as a motion vector of the integrated block image, and when the difference value exceeds a predetermined threshold, the integrated block The image integration is canceled, and a motion vector for each block image before the integration is output.

  Further, in the motion vector detection device according to the present invention, the motion vector detection unit calculates a power ratio in a high frequency region of the spatial frequency from the spatial frequency spectrum, and the power ratio in the high frequency region of the spatial frequency is calculated. A motion search range determination unit that sets a motion search range narrower as it is higher, and a block matching unit that detects a motion vector for each block image generated by the block division unit in the motion vector search range. Features.

Furthermore, in the motion vector detection device according to the present invention, the motion vector detection unit may increase the block size and the size of the motion search range as the power ratio in the high-frequency region in the time direction over a plurality of frames in the original image increases. Or a resolution table hierarchically defined such that the larger the ratio of power in the low-frequency region in the spatial direction of one frame in a moving image, the larger the block size and the larger the motion search range. A resolution determination unit that determines a resolution value by detecting a power ratio of the high frequency region in the time direction or a power ratio of the low frequency region in the spatial direction;
With reference to the table, motion vector detection is started from the hierarchy of the block size and the motion search range corresponding to the resolution value, and the block size and the motion search range are gradually smaller than the block size. A hierarchical motion vector detection unit that shifts to motion vector detection in a hierarchy having a size of a motion search range smaller than that.

  In order to solve the above problem, an encoding apparatus according to the present invention includes the above-described motion vector detection apparatus, and includes the original image, motion vector information and motion detection block information output from the motion vector detection unit. It is characterized by generating encoded data obtained by encoding.

  In order to solve the above problem, a decoding device according to the present invention acquires the original image, the motion vector information, and the encoded data of the motion detection block information from the encoding device described above, and the original image The decoded image is generated.

  In order to solve the above problems, a program according to the present invention causes a computer to function as the above-described motion vector detection device, encoding device, or decoding device.

  According to the present invention, image features are determined for each block of the original image, and blocks that capture the same object are integrated, so that the block size is optimized to reduce the amount of motion vector information and high accuracy. Motion vector information can be obtained.

It is a block diagram which shows the structure of the motion vector detection apparatus of Example 1 by this invention. It is a block diagram which shows the structure of the motion vector detection part in the motion vector detection apparatus of Example 1 by this invention. It is a block diagram which shows the structure of the motion vector optimization part in the motion vector detection apparatus of Example 1 by this invention. It is a figure which illustrates the block set in the average brightness | luminance difference calculation part of the motion vector detection apparatus of Example 1 by this invention. It is a figure explaining the process in the block size correction part of the motion vector detection apparatus of Example 1 by this invention. It is a flowchart which shows the operation | movement in the motion vector correction part of the motion vector detection apparatus of Example 1 by this invention. It is a figure which shows an example of the motion vector for every block in a block set input into the motion vector correction part of the motion vector detection apparatus of Example 1 by this invention. It is a block diagram which shows the structure of the encoding apparatus of Example 1 by this invention. It is a block diagram which shows the structure of the decoding apparatus of Example 1 by this invention. It is a block diagram which shows the structure of the super-resolution auxiliary information generation device of Example 1 by this invention. It is a block diagram which shows the structure of the motion vector detection apparatus of Example 2 by this invention. It is a block diagram which shows the structure of the motion vector detection part in the motion vector detection apparatus of Example 2 by this invention. It is a block diagram which shows the structure of the time direction high frequency area | region power calculation part of the motion vector detection part in the motion vector detection apparatus of Example 2 by this invention. It is a block diagram which shows the structure of the spatial direction low frequency area | region power calculation part of the motion vector detection part in the motion vector detection apparatus of Example 2 by this invention. It is a figure which illustrates the relationship between the ratio of the power of the low frequency area | region of the spatial direction of the motion vector detection part of the motion vector detection apparatus of Example 2 by this invention, and a motion detection start floor. It is a block diagram which shows the structure of the hierarchical motion vector detection part of the motion vector detection part in the motion vector detection apparatus of Example 2 by this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram of a motion vector detection apparatus according to a first embodiment of the present invention. As shown in FIG. 1, the motion vector detection device 1 includes a block division unit 10, a spatial frequency analysis unit 20, a motion vector detection unit 30, and a motion vector optimization unit 40.

  The block division unit 10 divides an input original image into blocks of a predetermined size (for example, 8 pixels × 8 lines), and a block image sequence in units of blocks is converted into a spatial frequency analysis unit 20, a motion vector detection unit 30, and The result is output to the motion vector optimization unit 40. The block size may be stored in a storage unit (not shown) and read.

  The spatial frequency analysis unit 20 performs a spatial frequency analysis on the divided image sequence (block image sequence) input from the block division unit 10 to generate a spatial frequency spectrum, and the motion vector detection unit 30 uses the spatial frequency spectrum for each block image. And output to the motion vector optimization unit 40. For the spatial frequency analysis, wavelet decomposition, discrete cosine transform (DCT), or the like is used.

  The motion vector detection unit 30 detects a motion vector between the base frame and the reference frame for each block image. FIG. 2 is a block diagram illustrating a configuration of the motion vector detection unit 30. As shown in FIG. 2, the motion vector detection unit 30 includes a motion search range determination unit 31 and a block matching unit 32.

  The motion search range determination unit 31 determines a motion search range that is a search range of a motion vector according to the spatial frequency spectrum input from the spatial frequency analysis unit 20, and stores the motion search range information indicating the motion search range as a block matching unit. 32. Specifically, the motion search range determination unit 31 calculates the power ratio of the high frequency region of the spatial frequency from the spatial frequency spectrum input from the spatial frequency analysis unit 20, and the power ratio of the high frequency region of the spatial frequency is high. In this case, the motion search range is set to be narrower because it is highly likely that the region is a still region or a motion region with a small amount of motion, and the motion search range is set wider as the power ratio in the high frequency region of the spatial frequency decreases. .

  The power ratio in the high frequency region of the spatial frequency is, for example, the power ratio in the high frequency region that exceeds half the sampling frequency. Further, the motion search range is set such that the maximum search range in the horizontal axis direction is [x1, x2], and the power ratio of the high frequency region in the horizontal direction is P_x, for example, the search range in the horizontal axis direction is (1-P_x). X (x2-x1). Similarly, in the vertical axis direction, the search range is set to [y1, y2] at the maximum, and the power ratio of the high frequency region in the horizontal direction is P_y. For example, the search range in the horizontal axis direction is (1−P_y) × (y2− y1).

  The block matching unit 32 detects, for each block, a motion vector with a small pixel precision within the motion search range set by the motion search range determination unit 31, and performs motion vector optimization on the detected motion vector information (alignment information). To the unit 40. A final motion vector is determined by a motion vector correction unit 44 described later.

  Specifically, the block matching unit 32 performs block matching on the reference frame image divided from the block input from the image dividing unit 10 using the original image of the reference frame of the reference frame as a reference frame image, and performs block matching. Each time, a motion vector between the base frame and the reference frame is detected. For block matching, for example, an SSD (Sum of Squared Difference) method or an SAD (Sum of Absolute Intensity Difference) method is used. In addition, block matching is performed with decimal pixel accuracy by interpolation processing using a parabolic fitting function shown in Equation (1), for example.

  Here, when the pixel position at the search position is x, SSD (x) represents the SSD value at the pixel position, and more specifically, SSD (0) is the center position (position where the SSD value is minimized). SSD (−1) is the SSD value at the position of the adjacent pixel in the −x direction or −y direction from the center position, and SSD (1) is the SSD value at the position of the adjacent pixel in the + x direction or + y direction from the center position. Represents. From equation (1), pixel positions (decimal pixel positions) with decimal pixel precision in the horizontal or vertical direction can be calculated respectively.

  The motion vector optimization unit 40 corrects the motion vector input from the motion vector detection unit 30 based on the divided image sequence input from the block division unit 10 and the spatial frequency spectrum input from the spatial frequency analysis unit 20. And optimize. FIG. 3 is a block diagram illustrating a configuration of the motion vector optimization unit 40. As shown in FIG. 3, the motion vector optimization unit 40 includes a spatial direction high-frequency region power calculation unit 41, an average luminance difference calculation unit 42, a block size correction unit 43, and a motion vector correction unit 44.

  The average luminance difference calculation unit 42 determines a block set image including a plurality of adjacent block images. FIG. 4 is a diagram illustrating a block set image. In FIG. 4, the block set image S is composed of a representative block image A that is shaded and three block images B, C, and D adjacent to the representative block image A. The number of blocks in the block set image S is not limited to four, and may be eight or nine, for example. The representative block is one block image in the block set image. In FIG. 4, the block image located at the upper left of the block set image is the representative block image, but the position of the representative block image is limited to the upper left. It is not a thing.

  The average luminance difference calculation unit 42 calculates the average luminance for each block image input from the block dividing unit 10, and then calculates the average luminance difference value between the block images for each block set image. The difference value is, for example, a sum of absolute values of differences, a product of absolute values of differences, or a sum of squares of differences. For the block set image S in FIG. 4, the average luminance of the block image A is Aave, the average luminance of the block image B is Bave, the average luminance of the block image C is Cave, the average luminance of the block image D is Dave, and the difference value is different. When the absolute value of the difference is taken, the difference value is | Aave−Bave | + | Aave−Cave | + | Aave−Dave | + | Bave−Cave | + | Bave−Dave | + | Cave−Dave | .

  The average luminance difference calculation unit 42 has a smaller average luminance difference value than the representative block image A when the average luminance difference value between the block images in the block set image S is equal to or less than a predetermined threshold. When the average luminance small difference flag FL_L (i, j) indicating ON is set to ON and the average luminance difference value between the block images in the block set image S exceeds a predetermined threshold value, the representative block image A Thus, the small average luminance difference flag FL_L (i, j) is set to OFF. Here, (i, j) is an index indicating the position of the block image in the frame image. Then, the average luminance difference calculation unit 42 outputs the average luminance small difference flag FL_L (i, j) for each block image to the block size correction unit 43.

  The spatial direction high frequency region power calculation unit 41 calculates the power ratio of the high frequency region of the spatial frequency for the representative block image from the spatial frequency spectrum for each block image input from the spatial frequency analysis unit 20, and calculates the calculated spatial frequency When the ratio of the power in the high frequency region is equal to or lower than a predetermined threshold, the spatial low frequency flag FL_S (i, j) indicating that the spatial frequency is low with respect to the representative block image is set to on (true), When the calculated power ratio in the high frequency region of the spatial frequency exceeds a predetermined threshold, the spatial low frequency flag FL_S (i, j) is set to off (false). Here, (i, j) is an index indicating the position of the block in the image. Then, the spatial direction high frequency region power calculation unit 41 outputs the spatial low frequency flag FL_S (i, j) for each block image to the block size correction unit 43.

  The block size correcting unit 43 includes a spatial low frequency flag FL_S (i, j) input from the spatial direction high frequency region power calculating unit 41 and a small average luminance difference flag FL_L (i, j) input from the average luminance difference calculating unit 42. Based on j), it is determined whether or not to integrate the block images in the block set image, and the size of the block image is corrected. FIG. 5 is a diagram for explaining the processing of the block size correcting unit 43. As illustrated in FIG. 5, the block size correcting unit 43 has the spatial low frequency flag FL_S (i, j) on and the small average luminance difference flag FL_L (i, j) on for the representative block image A. In this case, the block images A, B, C, and D in the block set image S are integrated into one integrated block image T. That is, the block size of the integrated block image T is the size of the block set S. The block size correcting unit 43 outputs block information indicating the block size of the corrected block image (block image or integrated block image) to the motion vector correcting unit 44. It should be noted that the block information may be a flag indicating whether or not each initial (before integration) block image is integrated.

  In other words, if the power of the spatial high-frequency area of the representative block is low, the block image can be an out-of-focus area in a stationary area or a moving area with a small amount of motion, or a moving area image with a large amount of motion and a large amount of motion blur. High nature. The block size correcting unit 43 considers integration with surrounding block images because image features cannot be captured with a small block size and the accuracy of motion vector detection is reduced. Then, when the average luminance difference with the surrounding block image is small, there is a high possibility that the same object as the surrounding block is captured. Thereby, the detection accuracy of a motion vector can be improved.

  The motion vector correction unit 44 corrects the motion vector from the motion vector information for each block image input from the block matching unit 32 and the block information input from the block size correction unit 43, and sends the corrected motion vector to the outside. In addition to outputting, motion detection block information indicating the block size of the image corresponding to the motion vector to be output is output to the outside. FIG. 6 is a flowchart showing the operation of the motion vector correction unit 44. FIG. 7 is a diagram illustrating an example of a motion vector for each block image in the block set image. The operation of the motion vector correction unit 44 will be described with reference to FIGS.

  First, the motion vector correction unit 44 determines whether or not the block images are integrated from the block information input from the block size correction unit 43 (step S11). If it is determined in step S11 that the block images are not integrated, the motion vector for each block image input from the block matching unit 32 is output as it is as the final motion vector (step S16). On the other hand, if it is determined in step S11 that the block images have been integrated, the motion vectors a, b, c, d of the respective blocks before integration in the integrated block image T are acquired (step S12). Next, a difference value between the motion vectors a, b, c, d acquired in step S12 is calculated (step S13). The difference value is, for example, a sum of absolute values of differences, a product of absolute values of differences, or a sum of squares of differences.

  Then, the motion vector correction unit 44 determines whether or not the difference value calculated in step S12 is equal to or less than a predetermined threshold (step S14). If it is determined in step S14 that the difference value is equal to or smaller than the predetermined threshold, the motion vectors are integrated, and the average value of the motion vectors a, b, c, d is output as the motion vector of the integrated block image T (step S15). On the other hand, if it is determined in step S14 that the difference value exceeds the predetermined threshold, the block image integration is canceled and a motion vector for each block image before integration is output (step S15). In addition, the motion vector correction unit 44 outputs information indicating the size of an image (block image or integrated block image) corresponding to the output motion vector to the outside as motion detection block information. Note that the motion detection block information may be a flag indicating whether or not each of the initial (pre-integration) block images has been integrated.

  When it is determined in step S11 that the block images have been integrated, the motion vector correction unit 44 acquires the motion vectors a, b, c, and d of each block before integration in the integrated block image. (Step S12) The average value of the motion vectors a, b, c, and d may be output as the motion vector of the integrated block image T without performing the processing of Step S13 and Step S14. However, in order to realize high accuracy of the motion vector, the processing of step S13 and step S14 is performed, and when the motion detection accuracy of each block image before integration is considered high, that is, the difference value of the motion vector is predetermined. It is preferable to integrate motion vectors only when the threshold value is equal to or less than the threshold value.

  When adjacent integrated block images exist and the motion vectors of the integrated block images are integrated in step S14, the motion vectors of the integrated block images are acquired, and the difference value of the acquired motion vectors is calculated. When the calculated difference value is equal to or smaller than a predetermined threshold, an image obtained by integrating the integrated block images is set as a new integrated block image, and an average value of the acquired motion vectors is output as a motion vector of the new integrated block image. May be.

  Note that a computer can be suitably used to function as the motion vector detection device 1 described above, and such a computer can store a program describing processing contents for realizing each function of the motion vector detection device 1. This program can be realized by reading out and executing this program by the CPU (central processing unit) of the computer.

  As described above, according to the motion vector detection device 1 and the program thereof, it is possible to optimize the block size, reduce the amount of motion vector information, and obtain highly accurate motion vector information.

[Encoding device]
Next, an encoding device using the motion vector detection device 1 described above will be described. FIG. 8 is a block diagram showing a configuration of the encoding apparatus according to the first embodiment of the present invention. As illustrated in FIG. 8, the encoding device 60 includes a motion vector detection device 1, a subtraction unit 61, a quantization unit 62, a variable length encoding unit 63, an inverse quantization unit 64, and an inverse orthogonal transform unit. 65, an adder 66, a frame memory 67, and a motion compensator 68.

  The subtraction unit 61 generates a difference image between the original image and the predicted image input from the motion compensation unit 68 and outputs the difference image to the motion vector detection device 1.

  When the motion vector detection device 1 is applied to the encoding device 60, the motion vector detection device 1 adds the spatial frequency spectrum generated by the spatial frequency analysis unit 20 in addition to the motion vector information and the motion detection block information (for example, The conversion coefficient generated by the discrete cosine transform is also output to the outside. The motion vector detection device 1 performs the above-described motion vector detection process on the difference image input from the subtraction unit 61, outputs a spatial frequency spectrum to the quantization unit 62, and provides motion vector information and motion detection block information. Is output to the variable length encoding unit 63 and the motion compensation unit 68.

  The quantization unit 62 performs a quantization process on the spatial frequency spectrum input from the motion vector detection device 1 and outputs the result to the variable length encoding unit 63 and the inverse quantization unit 64.

  The variable length coding unit 63 performs variable length coding processing on the spatial frequency spectrum quantization data input from the quantization unit 62 and the motion vector information and motion detection block information input from the motion vector detection device 1. To generate a bit stream of encoded data and output it to the decoding side.

  The inverse quantization unit 64 performs inverse quantization processing on the quantized data of the spatial frequency spectrum input from the quantization unit 62 and outputs the result to the inverse orthogonal transform unit 65.

  The inverse orthogonal transform unit 65 performs inverse orthogonal transform on the spatial frequency spectrum input from the inverse quantization unit 64 (for example, the discrete cosine transform has been performed by the spatial frequency analysis unit 20 of the motion vector detection device 1). In this case, inverse discrete cosine transform (IDCT) is performed and output to the adder 66.

  The adding unit 66 adds the image input from the inverse orthogonal transform unit 65 and the predicted image obtained from the motion compensation unit 68 to generate a decoded image, and outputs the decoded image to the frame memory 67.

  The motion compensation unit 68 performs motion compensation on the reference frame image stored in the frame memory 67 using the motion vector information and the motion detection block information input from the motion vector detection device 1 to generate a predicted image, The predicted image is output to the subtraction unit 61.

  Note that a computer can be suitably used to cause the above-described encoding device 60 to function, and such a computer stores a program describing processing contents for realizing each function of the encoding device 60 in the storage of the computer. The program can be realized by reading out and executing the program by the CPU of the computer.

  As described above, according to the encoding device 60 and its program, it is possible to encode and transmit an optimized high-precision motion vector.

[Decoding device]
Next, a decoding device that decodes the data encoded by the encoding device 60 described above will be described. FIG. 9 is a block diagram showing the configuration of the decoding apparatus according to the first embodiment of the present invention. As illustrated in FIG. 9, the decoding device 70 includes a variable length decoding unit 71, an inverse quantization unit 72, an inverse orthogonal transform unit 73, an addition unit 74, a frame memory 75, and a motion compensation unit 76. .

  The variable length decoding unit 71 performs variable length decoding processing on the encoded data input from the encoding device 60, outputs the quantized data of the spatial frequency spectrum to the inverse quantization unit 72, and provides motion vector information and motion The detected block information is output to the motion compensation unit 76.

  The inverse quantization unit 72 performs inverse quantization processing on the quantized data of the spatial frequency spectrum input from the variable length decoding unit 71 and outputs the result to the inverse orthogonal transform unit 73.

  The inverse orthogonal transform unit 73 performs inverse orthogonal transform (for example, IDCT) on the spatial frequency spectrum input from the inverse quantization unit 72 and outputs the obtained image to the addition unit 74.

  The adder 74 adds the image input from the inverse orthogonal transform unit 73 and the predicted image input from the motion compensation unit 76 to restore the image, and outputs the decoded image to the frame memory 75 and the outside.

  The motion compensation unit 76 performs motion compensation on the reference frame image stored in the frame memory 75 using the motion vector information input from the variable length decoding unit 71 to generate a prediction image, and adds the prediction image to the addition unit. Output to 74.

  Note that a computer can be suitably used to cause the above-described decryption apparatus 70 to function, and such a computer stores a program describing processing contents for realizing each function of the decryption apparatus 70 in a storage unit of the computer. This can be realized by storing the program and executing it by the CPU of the computer.

  As described above, according to the decoding device 70 and the program thereof, a high-quality image can be decoded from the motion vector information and the motion detection block information.

[Super-resolution auxiliary information generator]
Next, the super-resolution auxiliary information generation device 80 using the motion vector detection device 1 described above will be described. FIG. 10 is a block diagram showing a configuration of the super-resolution auxiliary information generating apparatus according to the first embodiment of the present invention. As shown in FIG. 10, the super-resolution auxiliary information generation device 80 includes a motion vector detection device 1 and an image reduction unit 81.

  The image reduction unit 81 inputs an original image sequence and outputs a reduced image sequence reduced at a predetermined reduction rate. FIG. 10 shows an example of generating a reduced image by wavelet decomposition, and the image reduction unit 81 includes a wavelet decomposition unit 811 and a reduced image generation unit 812.

  The wavelet decomposition unit 811 performs image reduction using an n-th order spatial wavelet decomposition method (n is a natural number including 0). The wavelet decomposition unit 811 reads out the mother wavelet information (that is, information on the filter coefficient indicating the decomposition coefficient of the mother wavelet) and the information on the image reduction ratio that are stored in advance in a storage unit (not shown), and the mother wavelet information Is used to perform wavelet decomposition at a resolution corresponding to the image reduction rate, and the wavelet decomposed image is output to the reduced image generation unit 812.

  The reduced image generation unit 812 extracts the low-frequency component of the wavelet-decomposed image (for example, when the first-order spatial wavelet decomposition is performed with n = 1, the image of the low-frequency component in the horizontal and vertical directions). Then, a reduced image corresponding to the extracted low frequency component is generated. In addition, since it is possible to control the aliasing distortion amount included in the reduced image sequence by changing the mother wavelet by changing the mother wavelet, in this case, the information on the variable mother wavelet is stored in the storage unit (see FIG. (Not shown).

  It should be noted that a computer can be suitably used to function as the above-described super-resolution auxiliary information generation device 80, and such a computer describes the processing contents for realizing each function of the super-resolution auxiliary information generation device 80. This program can be realized by storing the program in a storage unit of the computer and reading and executing the program by the CPU of the computer.

  A high-resolution super-resolution image can be generated using the motion vector, the motion detection block information, and the plurality of reduced images output from the super-resolution auxiliary information generation device 80. Here, when the motion vector difference between the block images before integration in the integrated block image is small by the motion vector correction unit 44 of the motion vector detection device 1, the average of the motion vectors of the block images before integration is integrated. Since the motion vector of the block image is used, the amount of calculation can be reduced when super-resolution is generated from a plurality of reduced images generated by the image reduction unit 81. In regions where the power ratio of the high frequency region of the spatial frequency is low, there is little effect on the image quality even without performing high-precision position adjustment, so high-quality super-resolution images while reducing motion vector information Can be generated.

  Next, the motion vector detection device 2 according to the second embodiment of the present invention will be described in detail with reference to the drawings. FIG. 11 is a block diagram of the motion vector detection apparatus according to the second embodiment of the present invention. As illustrated in FIG. 11, the motion vector detection device 2 includes a block division unit 10, a spatial frequency analysis unit 20, a motion vector detection unit 50, and a motion vector optimization unit 40. The motion vector detection device 2 according to the second embodiment is different from the motion vector detection device 1 according to the first embodiment in that a motion vector detection unit 50 is provided instead of the motion vector detection unit 30.

  FIG. 12 is a block diagram illustrating a configuration of the motion vector detection unit 50. As shown in FIG. 12, the motion vector detection unit 50 includes a resolution determination unit 56 and a hierarchical motion vector detection unit 55. The block size of the divided image input to the motion vector detection unit 50 is, for example, the maximum value of the motion search range (for example, 256 pixels × 256 lines).

  The resolution determination unit 56 hierarchically defines the resolution so that the larger the ratio of the power in the high-frequency region in the time direction over a plurality of frames in the original image is, the larger the block size and the larger the motion search range are, and / or the moving image. The table of resolution defined hierarchically so as to have a larger block size and a larger motion search range as the proportion of power in the low-frequency region in the spatial direction of one frame in FIG. The resolution value is determined by detecting the power ratio of the region and / or the power ratio of the low frequency region in the spatial direction. Specifically, the resolution determination unit 56 includes a time direction high frequency domain power calculation unit 51, a spatial resolution rank determination unit 52, a spatial direction low frequency domain power calculation unit 53, and a motion detection start resolution determination unit 54. .

Temporal high-frequency range power calculation unit 51 includes a reference frame F to be used for reference frame F (t _C) and the motion search to detect the motion vector (t _R), a plurality of frames at time t = t 0 _... t _m F (t ₀ ),..., F (t _C ),..., F (t _R ),..., F (t _m ) are input, and all pixels in the divided image F (t _C ) of the reference frame are input. Then, after decomposing the plurality of frames into a frequency region of the maximum rank specified in advance in the time direction, the power for each frequency band in the time direction in all pixels is calculated, and the calculated power for each frequency band in the time direction in all pixels is calculated. The ratio of the power in the high frequency region in the time direction is calculated and output to the spatial resolution rank determining unit 52.

  FIG. 13 is a block diagram showing a configuration of the time direction high frequency domain power calculation unit 51. As shown in FIG. 13, the time direction high frequency domain power calculation unit 51 includes a time direction one-dimensional Nmax-order discrete wavelet decomposition processing unit 511 and a time direction frequency band-specific power calculation unit 512.

The time direction one-dimensional Nmax-order discrete wavelet decomposition processing unit 511 performs a base frame split image F (t _C ) (hereinafter referred to as a base split image) and a reference frame split image F (t _R ) (hereinafter a reference split image). A divided image sequence at time t = t ₀ ... T _m is input, and Nmax-order (for example, 4th-order) discrete wavelet decomposition in the time direction is performed on all pixels of the reference divided image.

  The power calculation unit 512 for each time direction frequency band calculates the power for each frequency band in the time direction in all pixels of the reference divided image, and calculates the power in the time direction high frequency region from the power for each frequency band in the time direction for all the calculated pixels. The power ratio is calculated and output to the spatial decomposition rank determining unit 52.

  The spatial resolution rank determination unit 52 has a larger dynamic region area as the proportion of power in the high frequency region in the time direction increases from the proportion of power in the high frequency region in the time direction calculated by the time direction high frequency region power calculation unit 51, and It is determined that the amount of motion is large, the spatial frequency decomposition rank Ns (that is, the resolution in the spatial direction) is determined according to the power ratio of the high frequency region in the time direction, and information on the determined spatial frequency decomposition rank Ns is stored in the space. It outputs to the direction low frequency area | region power calculation part 53. FIG.

  For example, as shown in Table 1, a table defined in advance between the power ratio in the high frequency region in the time direction and the resolution rank Ns of the spatial frequency is held in advance. The high frequency region in the time direction is, for example, a range exceeding 1/2 of the sampling frequency.

  The spatial direction low frequency region power calculation unit 53 receives the reference divided image and the reference divided image and information on the spatial frequency decomposition rank Ns, and based on the spatial frequency decomposition rank Ns, the reference divided image and / or the reference divided image The spatial Ns-order discrete wavelet decomposition is performed on the image, the power for each spatial frequency band is calculated, the power ratio in the low frequency region in the spatial direction is calculated from the calculated power for each spatial frequency band, and the calculated space The ratio of the power in the low frequency region in the direction and the data obtained by spatial Ns-order discrete wavelet decomposition are output to the motion detection start resolution determination unit 54.

  Note that performing spatial Ns-order discrete wavelet decomposition on both the standard divided image and the reference divided image is a subsequent process when performing hierarchical motion vector detection with a fixed block size and a search range size. By performing the wavelet reconstruction in a hierarchical manner, it is advantageous in that a hierarchical motion vector detection with a variable block size and a search range size can be performed on the original image. In order to determine the resolution to be performed hierarchically, it is preferable to select the larger one of the ratio of the power in the low frequency region in the spatial direction from the standard divided image and the reference divided image. In the following description, an example will be described in which spatial Ns-order discrete wavelet decomposition is performed on both the standard divided image and the reference divided image.

  FIG. 14 is a block diagram illustrating a configuration of the spatial direction low frequency region power calculation unit 53. As shown in FIG. 14, the spatial direction low frequency region power calculation unit 53 includes a spatial direction two-dimensional Ns-order discrete wavelet decomposition processing unit 531 and a spatial direction frequency band-specific power calculation unit 532.

  The spatial direction two-dimensional Ns-order discrete wavelet decomposition processing unit 531 inputs the standard divided image and the reference divided image, and based on the decomposition rank Ns of the spatial frequency determined by the spatial decomposition rank determining unit 52, the standard divided image and the reference divided image Spatial Ns-order discrete wavelet decomposition is performed on all the pixels of the image.

  The power calculation unit 532 for each spatial frequency band calculates the power for each spatial frequency band in each of the standard divided image and the reference divided image, and uses the calculated power for each spatial frequency band in each of the standard divided image and the reference divided image. The ratio of the power in the low frequency region in the spatial direction is calculated, and information on the larger power ratio in the low frequency region in the spatial direction in each of the calculated standard divided image and reference divided image is sent to the motion detection start resolution determination unit 54. Output.

  The motion detection start resolution determination unit 54 determines that the power ratio of the low frequency region in the spatial direction is larger as the ratio of the power of the low frequency region in the spatial direction is larger from the information of the power ratio of the low frequency region in the spatial direction calculated by the spatial direction low frequency region power calculation unit 53 The resolution for starting motion vector detection hierarchically (hereinafter referred to as “motion detection start resolution”) is determined as the dynamic region area is large and the amount of motion is large as the power ratio of the high frequency region in the direction is small. The motion detection start resolution is determined according to the ratio of the power in the low frequency region in the spatial direction, and information on the determined hierarchical motion detection start resolution is obtained. The result is output to the hierarchical motion vector detection unit 55. For example, the motion detection start resolution determination unit 54 increases the number of floors at which motion vector detection is started (hereinafter referred to as “motion detection start floor”) ns as the power ratio in the low frequency region in the spatial direction increases. As described above, the motion detection start floor ns is determined according to the power ratio of the low frequency region in the spatial direction, and information on the determined motion detection start floor ns is output to the hierarchical motion vector detection unit 55. However, ns ≦ Ns.

  For example, as shown in Table 2, a table defined between the power ratio of the low frequency region in the spatial direction and the motion detection start resolution (or motion detection start floor ns) is held in advance. The low frequency region in the spatial direction is, for example, a range that is 1/2 or less of the frame frequency. As the motion detection start floor number ns increases, this means that the original image has a lower resolution, and the block size and the size of the motion search range increase relative to the original image. I mean. Here, for example, the spatial resolution of 1/16 means a reduction in resolution in which 16 pixels are sampled as one pixel at the horizontal sampling frequency Hs and the vertical sampling frequency Vs in the original image.

That is, as shown in FIG. 15, the motion detection start rank ns can be associated with the power ratio in the low frequency region in the spatial direction. For example, when Ns = 4, the respective powers of the low frequency region (LL ⁴ ) and the high frequency region (other than LL ⁴ ) are calculated, and the ratio of the low frequency region (LL ⁴ ) in the whole is 99.5% or more. If so, it is possible to reconstruct an image of only the low-frequency region of the four layers with the motion detection start rank ns = 4 (see FIG. 15D). Similarly, if the ratio of the low frequency region (LL ⁴ ) in the whole is 98.0% or more and less than 99.5%, an image of only three layers of the low frequency region (LL ³ ) can be reconstructed (FIG. 15). (C)), if the ratio of the low frequency region (LL ⁴ ) in the whole is 95.0% or more and less than 98.0%, it is possible to reconstruct an image of only the low frequency region (LL ² ) of two layers. (See FIG. 15 (b)). If the ratio of the low frequency region (LL ⁴ ) in the whole is less than 95.0%, an image of only one layer of the low frequency region (LL ¹ ) may be reconstructed. (See FIG. 15 (a)).

  The hierarchical motion vector detection unit 55 generates a reference divided image and a reference divided image in the spatial direction corresponding to the motion detection start rank ns in order to generate images of the reference divided image and the reference divided image in the spatial direction 2. Reconstruct the spatial ns-order wavelet according to the motion detection start rank ns for the dimension Ns-order discrete wavelet-decomposed data, perform motion vector detection with a predetermined block size and search range size, The space ns-1 floor wavelet is reconstructed, and motion vector detection is performed again with the predetermined block size and the size of the search range, and the highest level (ie, the original image level) The floor number is decremented and repeated until motion vector detection is performed with the determined block size and the size of the search range.

  FIG. 16 is a block diagram illustrating a configuration of the hierarchical motion vector detection unit 55. As shown in FIG. 16, the hierarchical motion vector detection unit 55 includes an ns-th order reconstructed motion vector detection unit 551 and a spatial first-order wavelet reconstruction unit 552.

  The ns-order reconstructed motion vector detection unit 551 generates a spatial ns-order wavelet according to the motion detection start rank ns to generate a reference divided image and a reference divided image in the low-frequency region in the spatial direction according to the motion detection start rank ns. Then, motion vector detection is performed by block matching with decimal pixel precision with a predetermined block size and search range size.

  The spatial first-order wavelet reconstruction unit 552 calculates the motion vector detection process by the spatial direction low-frequency region power calculation unit 53 in order to decrement and repeat the rank until the original image level corresponding to the highest rank is reached. The spatial Ns-order discrete wavelet decomposition data is subjected to wavelet reconstruction of the first floor higher in the spatial direction so that an image having a higher rank than the motion detection start rank ns is performed, and the ns-order reconstructed motion vector detection unit 551 Output. Therefore, the ns-order reconstructed motion vector detection unit 551 hierarchically performs motion vector detection processing using the reconstructed image of the reference frame divided image and the reference frame divided image obtained from the spatial first-order wavelet reconstruction unit 552. Repeat to determine and output the final motion vector.

  It is preferable that the motion vector detection is performed only with the highest rank (that is, the first floor) using the block matching method of the decimal pixel position by quadratic function approximation. For example, the decimal pixel position is calculated by parabolic fitting shown in Expression (1).

  The motion vector detection device 2 according to the second embodiment configured as described above can be applied to an encoding device, a decoding device, and a super-resolution device as in the first embodiment.

  Note that a computer can be suitably used to function as the motion vector detection device 2 described above, and such a computer can store a program that describes processing contents for realizing each function of the motion vector detection device 2. This program can be realized by reading out and executing the program by the CPU of the computer.

  As described above, according to the motion vector detection device 2 and the program thereof according to the second embodiment, the motion vector detection unit 50 performs hierarchical motion vector detection by sequentially repeating through spatial ns-order wavelet decomposition and reconstruction. It is not necessary to prepare the block size and the size of the search range that should be sequentially changed according to the hierarchy, and can be fixed, and the motion vector detection according to the motion detection start rank ns according to the image scene is performed, Since the motion vector optimizing unit 40 performs optimization based on the motion vector, the motion vector can be made more accurate.

  Although the above-described embodiments have been described as representative examples, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes may be made without departing from the scope of the claims. Is possible. For example, in the embodiment, downsampling is performed by wavelet decomposition, but it is not limited to wavelet decomposition as long as octave decomposition is performed.

  Since the present invention can obtain highly accurate motion vector information, it is useful for any application that performs image processing (eg, encoding processing, decoding processing, super-resolution image generation processing) using motion vector information. .

DESCRIPTION OF SYMBOLS 1, 2 Motion vector detection apparatus 10 Block division part 20 Spatial frequency analysis part 30, 50 Motion vector detection part 31 Motion search range determination part 32 Block matching part 40 Motion vector optimization part 41 Spatial direction high frequency area power calculation part 42 Average brightness | luminance Difference calculation unit 43 Block size correction unit 44 Motion vector correction unit 51 Time direction high frequency region power calculation unit 52 Spatial decomposition rank determination unit 53 Spatial direction low frequency region power calculation unit 54 Motion detection start resolution determination unit 55 Hierarchical motion vector detection unit 56 Resolution determination unit 511 Time direction one-dimensional Nmax-order discrete wavelet decomposition processing unit 512 Time direction frequency band-specific power calculation unit 531 Spatial direction two-dimensional Ns-order discrete wavelet decomposition processing unit 532 Spatial direction frequency band-specific power calculation unit 551 ns Composition movement Couttle detection unit 552 Spatial first-order wavelet reconstruction unit 61 Subtraction unit 62 Quantization unit 63 Variable length encoding unit 64, 72 Inverse quantization unit 65, 73 Inverse orthogonal transformation unit 66, 74 Addition unit 67, 75 Frame memory 68, 76 Motion compensation unit 71 Variable length decoding unit

Claims (9)

A motion vector detection device for detecting a motion vector between a base frame and a reference frame of an image,
A block dividing unit that generates a block image obtained by dividing an original image into blocks of a predetermined size;
A spatial frequency analysis unit that generates a spatial frequency spectrum by performing spatial frequency analysis on the block image;
A motion vector detection unit that detects a motion vector between a base frame and a reference frame for each block image;
A plurality of adjacent block images are set as block set images, and an average luminance difference calculating unit that calculates a difference value of average luminance between the block images in the block set image;
From the spatial frequency spectrum, a spatial direction high frequency region power calculation unit that calculates a power ratio of a high frequency region of a spatial frequency for one block image in the block set image;
For the block set image, when the difference value of the average luminance is less than or equal to a predetermined threshold and the power ratio of the high frequency region is less than or equal to the predetermined threshold, the block images in the block set image are integrated. A block size correction unit to be an integrated block image,
For the integrated block image, the average value of the motion vectors of each block image is output as the motion vector of the integrated block image, and for the non-integrated block image, for each block image detected by the motion vector detection unit A motion vector correction unit that outputs a motion vector and outputs motion detection block information indicating a block size of an image corresponding to the output motion vector;
A motion vector detection device comprising:   The motion vector correction unit calculates, for the integrated block image, a difference value of motion vectors of each block image before integration in the integrated block image, and when the difference value is equal to or less than a predetermined threshold, The average value of the motion vectors of each block image before integration is output as a motion vector of the integration block image, and if the difference value exceeds a predetermined threshold, the integration of the integration block image is canceled and the block before integration The motion vector detection apparatus according to claim 1, wherein a motion vector for each image is output. The motion vector detection unit calculates a power ratio in a high frequency region of a spatial frequency from the spatial frequency spectrum, and a motion search range in which the motion search range is set narrower as the calculated power ratio in the high frequency region of the spatial frequency is higher A decision unit;
A block matching unit for detecting a motion vector for each block image generated by the block dividing unit in the motion vector search range;
The motion vector detection device according to claim 1, further comprising: The motion vector detection unit
The resolution defined hierarchically so that the larger the power ratio of the high-frequency region in the time direction over a plurality of frames in the original image is, the larger the block size and the larger the motion search range is, or the lower the spatial direction of one frame in the moving image is A table of resolutions defined hierarchically so that the larger the frequency domain power ratio is, the larger the block size and the larger the motion search range is,
A resolution determination unit that determines a resolution value by detecting a power ratio of the high-frequency region in the time direction or a power ratio of the low-frequency region in the spatial direction;
With reference to the table, motion vector detection is started from the hierarchy of the block size and the motion search range corresponding to the resolution value, and the block size and the motion search range are gradually smaller than the block size. A hierarchical motion vector detection unit that shifts to motion vector detection in a hierarchy of a size of a motion search range smaller than
The motion vector detection device according to claim 1, further comprising:   5. A coding comprising the motion vector detection device according to claim 1, wherein the original image and motion vector information and motion detection block information output from the motion vector detection unit are encoded. An encoding apparatus for generating data.   6. The decoding apparatus according to claim 5, wherein the original image, the information on the motion vector, and the encoded data of the motion detection block information are acquired from the encoding apparatus according to claim 5, and a decoded image of the original image is generated. .   The motion vector detection program for functioning a computer as a motion vector detection apparatus as described in any one of Claims 1-4.   An encoding program for causing a computer to function as the encoding device according to claim 5.   A decoding program for causing a computer to function as the decoding device according to claim 6.

Images