JP2005252593A - Motion vector searching apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving vector retriever which determines a motion vector having a high compression efficiency, without making the quantity of calculation increase. <P>SOLUTION: An original image sub-sample filter 13 filters an original image read out by an original image memory controller 12 based on an original image sub-sampling rate, and an estimated image sub-sample filter 16 filters an estimated image read out by an estimated image memory controller 15 based on an estimated image sub-sampling rate. A plurality of pixel comparators 17 obtain the correlation values between the original and estimated images. A result decider 18 calculates the total value of the correlation values, to determine a retrieval position with the least the total value calculated within a retrieval range as a motion vector, and a controller 19 determines the original sub-sampling rate, the estimated image sub-sampling rate and the retrieval range, while taking into account of the number of the pixel comparators 17, based on coding control information of a coding system. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a motion vector search apparatus that searches for a motion vector in inter-frame predictive coding of a moving image.

  When encoding a moving image, inter-frame predictive encoding is generally performed in which only the difference between the original image and the predicted image on the time axis and the predicted image on the time axis is encoded in order to increase the compression rate. Yes. However, in order to find a search point with the smallest difference from a certain part of the original image, that is, a search point with high compression efficiency from the entire screen or a wide search range, a very large amount of calculation is required. It is difficult to reduce the size and power consumption. Therefore, the conventional motion vector search device reduces the amount of calculation by narrowing down the search range, or by thinning out pixels of the original image or predicted image, that is, sub-sampling, thereby searching a somewhat wide search range with a small amount of calculation. A motion vector search device that can be used is realized. For example, as shown in Patent Document 1, a method of further reducing the amount of calculation by making the subsample rate variable has been proposed.

JP-A-9-18882 (paragraph 0024)

  Since the conventional motion vector search apparatus is configured as described above, even if the calculation amount can be reduced by fixing or changing the sub-sample rate, the search range is fixed, so the compression efficiency is not necessarily high. There is a problem that it cannot be determined that a motion vector having high compression efficiency cannot be determined.

  The present invention has been made to solve the above-described problems. By changing the sub-sample rate and the search range according to the image to be encoded, the movement with high compression efficiency can be achieved without increasing the amount of calculation. An object of the present invention is to obtain a motion vector search device capable of determining a vector.

  The motion vector search device according to the present invention is based on an original picture memory that stores an original picture to be encoded, an original picture memory control unit that reads out an original picture at a designated original picture position from the original picture memory, and a designated original picture subsample rate. The original picture sub-sample filter for filtering the original picture read out by the original picture memory control unit, the predicted picture memory for storing the predicted picture to be searched, and the predicted picture at one or more specified search positions A predicted image memory control unit that reads from the predicted image memory; a predicted image subsample filter that filters the predicted image read by the predicted image memory control unit based on the instructed predicted image subsample rate; and the filter process A plurality of pixel comparators for obtaining a correlation value between the original image and the filtered predicted image, and each pixel comparator A result determination unit that calculates the total value of the correlation values and determines a search position having the smallest total value calculated within the search range as a motion vector, the original subsample rate based on the encoding control information of the encoding method, The predicted image subsample rate and the search range are determined in consideration of the number of pixel comparators, and the original image position, the determined search position within the search range, and the determined original image subsample rate and the predicted image sub And a control unit for instructing the sample rate.

  The present invention searches a wider search range with the same processing amount by determining an original image subsample rate, a predicted image subsample rate, and a search range based on encoding control information in consideration of the number of pixel comparators. It is possible to search for a motion vector with high compression efficiency without increasing the amount of calculation.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing the configuration of a motion vector search apparatus according to Embodiment 1 of the present invention. This motion vector search apparatus includes an original image memory 11, an original image memory control unit 12, an original image subsample filter unit 13, a predicted image memory 14, a predicted image memory control unit 15, a predicted image subsample filter unit 16, and a plurality of pixel comparison units 17. The result determination unit 18 and the control unit 19 are provided.

  The original image memory 11 stores the original image to be encoded, and the original image memory control unit 12 receives the original image position to be encoded from the control unit 19 and reads the original image from the original image memory 11. The original image subsample filter unit 13 receives one or a plurality of original image subsample rates corresponding to the original image position from the control unit 19 and performs filter processing. The predicted image memory 14 stores a predicted image to be searched, and the predicted image memory control unit 15 receives one or a plurality of search positions from the control unit 19 and reads the predicted image from the predicted image memory 14. The prediction image subsample filter unit 16 receives one or a plurality of prediction image subsample rates corresponding to the prediction position from the control unit 19 and performs a filter process.

  The plurality of pixel comparison units 17 output the correlation values of the original image and the predicted image, and the result determination unit 18 obtains the total value of the correlation values output from the plurality of pixel comparison units 17, for example, the sum, etc. A determination flag output when all the searches in the search range are completed is input, and a search position having the highest correlation in the search range is determined as a motion vector. The control unit 19 determines the feature of the image based on the encoding control information defined by the encoding method input from the outside, and determines the original image subsample rate, the predicted image subsample rate, and the search range according to the image feature. The number of pixel comparison units 17 is determined so that the plurality of pixel comparison units 17 can be used effectively in a predetermined processing cycle, and the original image position to be encoded, the subsample rate, the search position of the predicted image, Outputs the subsample rate and determination flag.

Next, the operation will be described.
FIG. 2 is a flowchart showing a process flow of the control unit 19. Step ST11 is a step of determining an original image position, Step ST12 is a step of determining an original image subsample rate, a predicted image subsample rate, and a search range based on the encoding control information, and Step ST13 is a search position within the determined search range. If the next search position is still within the search range, step ST14 returns to step ST13, and if all the processes for the search position within the search range are completed, step ST15 proceeds to step ST15. This is a step of outputting a determination flag for instructing determination of a motion vector. When the search for one original image position is completed, the process returns to step ST11 to search for the next original image position.

  FIG. 3 is a diagram showing an example of a subsampling method in units of 16 × 16 pixels. FIG. 3A shows a full sample using all the pixels as they are, and FIG. A quarter subsample thinned out by 4 units is shown.

  In step ST <b> 11 of FIG. 2, the control unit 19 determines an original image position to be encoded and outputs it to the original image memory control unit 12. At this time, the control unit 19 determines the original image position so as to move sequentially from the upper left of the screen to the right, for example. When the original image position is allocated in units of 16 × 16 pixels, for example, the original image memory control unit 12 that has received the original image position reads data for 16 × 16 pixels from the original image memory 11 and outputs the data to the original image subsample filter unit 13.

  In step ST12, the control unit 19 determines the characteristics of the image based on the encoding control information such as a video sequence (M value), f-code, picture type, etc. defined in, for example, the MPEG encoding method. The original image subsample rate, the predicted image subsample rate, and the search range are determined according to the characteristics in consideration of the number of pixel comparison units 17 so that the plurality of pixel comparison units 17 can be used effectively in a predetermined processing cycle. The determined original image subsample rate is output to the original image subsample filter unit 13, and the determined predicted image subsample rate is output to the predicted image subsample filter unit 16. Here, when the feature of the image determined based on the encoding control information indicates, for example, a fast moving image, the control unit 19 increases the sub-sample rate and thins out many pixels to set the search range. If the image is wide and shows a slow and complicated image, the sub-sample rate is lowered, for example, the full sub-sample rate is set to narrow the search range.

  The original image subsample filter unit 13 that has received the original image of 16 × 16 pixels from the original image memory control unit 12 filters the original image of 16 × 16 pixels based on the original image subsample rate received from the control unit 19, and all the pixel comparison units 17. One pixel at a time. For example, when there are 16 × 16 pixels (256 pixels) and the subsample rate is the full sample shown in FIG. 3A, each pixel data of 16 × 16 pixels (256 pixels) is one by one. Assigned to. Further, when the subsample rate is 1/4 subsample shown in FIG. 3B, the number of pixel data is 4 × 16 pixels (64 pixels), so that the pixel data for 64 pixels is first compared with 64 pixels. Similarly, the pixel data for the same 64 pixels are allocated to the other 64 pixel comparison units 17. In this way, the same 64 pieces of pixel data are assigned to four sets of 64 pieces of pixel comparison units 17.

  In step ST13, the control unit 19 determines a search position within the search range determined in step ST12 and outputs the search position to the predicted image memory control unit 15. At this time, the control unit 19 determines to move the search position in order, for example, from the upper left of the search range to the right. When the search position is allocated in units of one pixel and the original image position is allocated in units of 16 × 16 pixels, for example, the predicted image memory control unit 16 that has received the search position sets the search position to the upper left pixel of the predicted image in units of 16 × 16 pixels, for example. A predicted image in units of 16 × 16 pixels is read out from the predicted image memory 14 and output to the predicted image subsample filter unit 16. At this time, when a plurality of search positions are output, the predicted image memory control unit 15 reads a plurality of predicted images from the predicted image memory 14 and outputs them to the predicted image subsample filter 16.

  The prediction image subsample filter unit 16 that has received the prediction image in units of 16 × 16 pixels from the prediction image memory control unit 15 filters the received prediction image in units of 16 × 16 pixels based on the prediction image subsample rate received from the control unit 19. , One pixel at a time is output to all the pixel comparison units 17. For example, when the predicted image subsample rate is the full sample shown in FIG. 3A, pixel data of 16 × 16 pixels (256 pixels) is assigned to each pixel comparison unit 17. When the subsample rate is ¼ subsample shown in FIG. 3B, a plurality of search positions are output from the control unit 19, and the number of pixel data after the subsample is 4 × 16 pixels (64 pixels). Therefore, four prediction images can be input to the 256 pixel comparison units 17. Accordingly, four search positions are input from the control unit 19 to the prediction image memory control unit 15, and the prediction image memory control unit 15 reads prediction images in units of 16 × 16 pixels at different search positions, and the prediction image sub-sample filter unit 16 has 4 A set of 64 pixel prediction images is generated and output to the pixel comparison unit 17.

  That is, when the predicted image sub-sample rate is a full sample, for example, a search of 16 × 16 pixels at a search position of −32 at the horizontal position or the vertical position can be processed in one cycle. In one cycle, four search positions, that is, −32, −16, 0, +16, which are horizontal positions or vertical positions, can be processed in one cycle. In this 1/4 subsample example, a search range of 4 times can be searched with the same processing amount. However, in the case of 1/8 subsample, an 8 times search range can be searched with the same processing amount, which is the same. This means that a wider search range can be searched with the processing amount.

  In this way, by making the subsample rate and search range variable based on the encoding control information, in the case of a fast moving image, the subsample rate is increased to thin out many pixels to widen the search range, In the case of complex images with slow motion, the sub-sample rate is reduced, for example, the full sub-sample rate is narrowed and the search range is narrowed to search in detail, thereby increasing the compression efficiency without increasing the amount of computation. You can get a vector.

  Each pixel comparison unit 17 calculates a correlation between the input pixel data of the original image and the pixel data of the predicted image, for example, a difference absolute value or a difference square value, and outputs the result to the result determination unit 18. The result determination unit 18 that has input the correlation value from each pixel comparison unit 17 calculates the total value of the correlation value, for example, the sum, and holds the total value and the search position in the case of the first search position, and searches from the second time. In the case of the position, the total value having the higher correlation and the search position are held, for example, the smallest total value compared to the held total value. In addition, when a plurality of sets of the same original image are assigned to the pixel comparison unit 17, the result determination unit 18 selects and holds the one having the highest correlation among the pair of the original image and the predicted image. It is held after comparing with the total value.

  In step ST14, the control unit 19 confirms whether or not the next search position is within the determined search range. If the next search position is still within the search range, the process returns to step ST13, and the control unit 19 predicts. The image memory control unit 15, the predicted image subsample filter unit 16, the pixel comparison unit 17, and the result determination unit 18 each perform the above-described processing, and when all processing is completed for the search position within the determined search range, Move on to ST15.

  In step ST15, the control unit 19 outputs a determination flag for instructing the determination of a motion vector to the result determination unit 18, and the result determination unit 18 outputs the search position held as a motion vector.

  As described above, according to the first embodiment, an original image subsample rate, a predicted image subsample rate, and a search range are effectively used based on encoding control information in a predetermined processing cycle. By deciding in consideration of the number of pixel comparison units 17 as possible, it is possible to search a wider search range with the same processing amount, and search for a motion vector with high compression efficiency without increasing the amount of calculation. The effect that it can be obtained.

Embodiment 2. FIG.
The block diagram showing the configuration of the motion vector search apparatus according to the second embodiment of the present invention is the same as FIG. 1 of the first embodiment. The motion vectors output from the result determination unit 18 for successive original image positions tend to have higher compression efficiency if they are finally oriented in the same direction during encoding. However, the direction of the searched motion vector often varies in flat images such as the sky and the sea. Thus, in the second embodiment, in the case of a flat image such as the sky or the sea, an offset is applied so that the searched motion vector is directed in the same direction.

  In FIG. 1, the result determination unit 18 calculates a total value, for example, a total value of correlation values output from the plurality of pixel comparison units 17, weights the calculated total value with a determination offset from the control unit 19, and performs control. The determination flag output when all the searches within the search range are completed from the unit 19 is input, and the motion having the highest correlation with the total value weighted by the determination offset and the search position is the smallest within the search range. Determine the vector. Further, the control unit 19 determines the characteristics of the image based on encoding control information such as a video sequence (M value), f-code, picture type, and the like, which are specified from the MPEG encoding method input from the outside, An original image sub-sample rate, a predicted image sub-sample rate, a search range, and a determination offset for directing a motion vector in a predetermined direction that can most effectively use the plurality of pixel comparison units 17 in a predetermined processing cycle according to image characteristics. Then, the original image position and the subsample rate to be encoded, the search position and subsample rate of the predicted image, the determination offset, and the determination flag are output. Other components are the same as those in the first embodiment.

Next, the operation will be described.
FIG. 4 is a flowchart showing a processing flow of the control unit 19. Step ST21 is a step of determining an original image position, step ST22 is a step of determining an original image subsample rate, a predicted image subsample rate, and a search range based on the encoding control information, and step ST23 is a search position within the determined search range. The step of determining and determining the determination offset based on the encoding control information, step ST24 returns to step ST23 if the next search position is still within the search range, and all processing is completed for the search position within the search range Step ST25 is a step for shifting to Step ST25, and Step ST25 is a step for outputting a determination flag for instructing the determination of a motion vector. When the search for one original image position is completed, the process returns to step ST21 to search for the next original image position.

  The process of the control unit 19 in step ST21 is the same as that of step ST11 in FIG. 2 of the first embodiment, and the process of the original picture memory control unit 12 is also the same as that of the first embodiment. Further, the process of the control unit 19 in step ST22 is the same as that of step ST12 in FIG. 2 of the first embodiment, and the process of the original picture sample filter unit 13 is also the same as that of the first embodiment.

  In step ST23, the control unit 19 determines the search position within the search range determined in step ST22 and outputs it to the predicted picture memory control unit 16, and directs the motion vector in a predetermined direction based on the encoding control information. Is determined and output to the result determination unit 18. Processing of the predicted image memory control unit 16 that has received the search position, processing of the predicted image subsample filter unit 16 that has received the predicted image in units of 16 × 16 pixels, and each pixel comparison unit that has received the pixel data of the original image and the pixel data of the predicted image The process 17 is the same as in the first embodiment.

  The result determination unit 18 that has input the correlation value from each pixel comparison unit 17 calculates the total value of the correlation value, for example, the sum, and calculates by adding the determination offset from the control unit 19 to the calculated total value, for example. The total value is weighted by the determination offset from the control unit 19, and in the case of the first search position, the weighted total value and the search position are retained, and in the case of the search position from the second time, the weighted total retained. Compared with the value, for example, the weighted total value having the higher correlation, such as the smallest weighted total value, and the search position are held. If the same original image is assigned to a plurality of sets and the pixel comparison unit 17, the weighted total value selected by selecting the one having the highest correlation among the sets of the original image and the predicted image. And then hold the comparison.

  The process of the control part 19 in step ST24 is the same as step ST14 of FIG. 2 of Embodiment 1. Further, the process of the control unit 19 in step ST25 is the same as step ST15 of FIG. 2 of the first embodiment, and the process of the result determination unit 18 is also the same as that of the first embodiment.

  As described above, according to the second embodiment, the same effects as those of the first embodiment can be obtained, and the correlation is determined with the determination offset for directing the motion vector in a predetermined direction determined based on the encoding control information. By weighting the total value of the values, the determined motion vector can be intentionally controlled. For example, in the case of variable-length encoding processing such as MPEG, it has the same amount and the same direction. The motion vectors can be continued, and an effect is obtained that a motion vector with higher compression efficiency can be searched without increasing the amount of calculation.

Embodiment 3 FIG.
5 is a block diagram showing a configuration of a motion vector search apparatus according to Embodiment 3 of the present invention. This motion vector search device is obtained by adding an image feature calculation unit 20 to the motion vector search device of FIG. 1 of the first embodiment, and the other configuration is the same as the configuration shown in FIG. The image feature calculation unit 20 calculates image features from the original image read from the original image memory 11 by the original image memory control unit 12 by, for example, detecting the average, variance, deviation, or scene change of pixel data. Further, the control unit 19 determines an original image subsample rate, a predicted image subsample rate, a search range, and a determination offset based on the encoding control information and the image feature calculated by the image feature calculation unit 20.

Next, the operation will be described.
The image feature calculation unit 20 calculates image features from the original image read by the original image memory control unit 12 by detecting, for example, the average, variance, and deviation of pixel data, and scene change that is a scene change point. Output to the control unit 20. The control unit 19 adds the image features calculated by the image feature calculation unit 20 to the image features determined based on the encoding control information, and the original image subsample rate, the predicted image subsample rate, the search range, and the determination Determine the offset. For example, for flat images that are not very complicated, such as the sky and the sea, the subsample rate is not reduced to search a wide range, and the judgment offset is adjusted to suppress motion vector variations, or images that move quickly In contrast, the sub-sample ratio is increased to search a wide range, and control to search for a motion vector with high compression efficiency is performed. In this case, as in the first embodiment, the control unit 19 can effectively use the plurality of pixel comparison units 17 in a predetermined processing cycle for the original image subsample rate, the predicted image subsample rate, and the search range. This is determined in consideration of the number of pixel comparison units 17. Other processes are the same as those in the second embodiment.

  As described above, according to the third embodiment, the same effects as those of the second embodiment can be obtained, and the image calculated by the average, variance, deviation, scene change detection, etc. of the pixel data of the original image can be obtained. By determining the original image subsample rate, the predicted image subsample rate, the search range, and the determination offset in consideration of the characteristics, it is possible to search for a motion vector with higher compression efficiency.

  In the third embodiment, the control unit 19 determines the original image subsample rate, the predicted image subsample rate, the search range, and the determination offset in consideration of the image features calculated by the image feature calculation unit 20. However, in consideration of the feature of the image calculated by the image feature calculation unit 20, only the original image subsample rate, the predicted image subsample rate, and the search range may be determined as in the first embodiment.

Embodiment 4 FIG.
FIG. 6 is a block diagram showing a configuration of a motion vector search apparatus according to Embodiment 4 of the present invention. This motion vector search device is obtained by adding a motion vector storage unit 21 to the motion vector search device of FIG. 1 of the first embodiment, and the other configuration is the same as the configuration shown in FIG. The motion vector storage unit 21 stores the motion vector output from the result determination unit 18 after the search is completed. The control unit 19 stores the motion vector stored in the motion vector storage unit 21 based on the encoding control information. The original image subsample rate, the predicted image subsample rate, the search range, and the determination offset are determined.

Next, the operation will be described.
The motion vector storage unit 21 stores a plurality of motion vectors output from the result determination unit 18. The control unit 19 determines the original image subsample rate, the predicted image subsample rate, the search range, and the determination offset based on the encoding control information and the stored motion vector. For example, in the case of an image in which the screen pans when the camera moves in the horizontal or vertical direction, everything on the screen appears to have moved in the same direction. In such a case, the motion vector has the same amount and the same direction. Therefore, the control unit 19 determines how much the image moves in which direction by analyzing the tendency of other motion vectors that have already been encoded, and the original image subsample rate corresponding to the image subsample rate, The predicted image subsample rate, search range, and determination offset can be determined. In this case, as in the first embodiment, the control unit 19 can effectively use the plurality of pixel comparison units 17 in a predetermined processing cycle for the original image subsample rate, the predicted image subsample rate, and the search range. This is determined in consideration of the number of pixel comparison units 17. Other processes are the same as those in the second embodiment.

  As described above, according to the fourth embodiment, the same effects as those of the second embodiment can be obtained, and the tendency of other motion vectors that have already been encoded is taken into account, and the original picture subsample rate and prediction By determining the image subsample rate, the search range, and the determination offset, it is possible to search for a motion vector with higher compression efficiency.

  In the fourth embodiment, the control unit 19 determines the original image subsample rate, the predicted image subsample rate, the search range, and the determination offset in consideration of the motion vectors stored in the motion vector storage unit 21. However, considering the motion vector stored in the motion vector storage unit 21, only the original image subsample rate, the predicted image subsample rate, and the search range may be determined in the same manner as in the first embodiment.

It is a block diagram which shows the structure of the motion vector search apparatus by Embodiment 1 of this invention. It is a flowchart which shows the flow of a process of the control part of the motion vector search apparatus by Embodiment 1 of this invention. It is a figure which shows an example of the subsampling method of a 16x16 pixel unit. It is a flowchart which shows the flow of a process of the control part of the motion vector search apparatus by Embodiment 2 of this invention. It is a block diagram which shows the structure of the motion vector search apparatus by Embodiment 3 of this invention. It is a block diagram which shows the structure of the motion vector search apparatus by Embodiment 4 of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 Original picture memory, 12 Original picture memory control part, 13 Original picture subsample filter part, 14 Predictive picture memory, 15 Predictive picture memory control part, 16 Predictive picture subsample filter part, 17 Pixel comparison part, 18 Result determination part, 19 Control part , 20 Image feature calculation unit, 21 Motion vector storage unit.

Claims (6)

An original picture memory for storing the original picture to be encoded;
An original image memory control unit for reading out the original image at the designated original image position from the original image memory;
An original image subsample filter that filters the original image read out by the original image memory control unit based on the instructed original image subsample rate;
A predicted image memory for storing a predicted image to be searched;
A predicted image memory control unit that reads a predicted image at one or more designated search positions from the predicted image memory;
A predicted image subsample filter that filters the predicted image read by the predicted image memory control unit based on the instructed predicted image subsample rate;
A plurality of pixel comparators for determining a correlation value between the filtered original image and the filtered predicted image;
A result determination unit that calculates a total value of correlation values obtained by the pixel comparators and determines a search position having the smallest total value calculated within the search range as a motion vector;
The original image subsample rate, the predicted image subsample rate, and the search range are determined in consideration of the number of pixel comparators based on the encoding control information of the encoding method, and the original image position and the determined search range are within And a control unit for instructing the determined original image subsample rate and the predicted image subsample rate. An image feature calculation unit that calculates the characteristics of the image of the original image to be encoded read by the original image memory control unit;
2. The motion vector search apparatus according to claim 1, wherein the control unit determines the original image subsample rate, the predicted image subsample rate, and the search range in consideration of the image features calculated by the image feature calculation unit. . A motion vector storage unit that stores the motion vector determined by the result determination unit;
2. The motion vector search apparatus according to claim 1, wherein the control unit determines an original image subsample rate, a predicted image subsample rate, and a search range in consideration of a motion vector stored in the motion vector storage unit. . The control unit determines a determination offset for directing the motion vector in a predetermined direction based on the encoding control information,
2. The motion vector search apparatus according to claim 1, wherein the result determination unit determines the motion vector by weighting a total value of correlation values with the determination offset. An image feature calculation unit that calculates the characteristics of the image of the original image to be encoded read by the original image memory control unit;
5. The motion according to claim 4, wherein the control unit determines an original image subsample rate, a predicted image subsample rate, a search range, and a determination offset in consideration of the image feature calculated by the image feature calculation unit. Vector search device. A motion vector storage unit that stores the motion vector determined by the result determination unit;
5. The motion vector search apparatus according to claim 4, wherein the control unit determines the original image subsample rate, the predicted image subsample rate, and the search range in consideration of the motion vectors stored in the motion vector storage unit. .

Images