JP2010258530A - Motion vector detecting apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a motion vector detecting apparatus for reducing the cost for a circuit and electric power. <P>SOLUTION: The cost for matching is calculated based on luminance pixel data from a search memory. The cost for memory access is calculated using color difference pixel data of the same area as the area where the cost for matching is calculated. Then, the total cost for matching and memory access is calculated, so that a vector indicating an area where the total value is minimum, and the size of the area are defined as a motion vector and the size of a reference area. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a motion vector detection device.

  In recent years, moving image processing technology has improved dramatically, and the use of digital images has become active. In encoding a moving image, inter-frame prediction is performed. In inter-frame prediction, the image data to be encoded is compared with the image data on the reference image, the difference is taken, and the motion vector indicating the position of the image data on the reference image referenced by the image data to be processed Is generated. When generating motion vectors, the image data that is similar to the image data to be processed is searched on the reference image, so access to the memory in which the reference image is stored frequently occurs, resulting in processing delay There is. Therefore, it is desired to improve the memory access efficiency.

  In a memory such as DDR3 (Double-Data-Rate3) that increases the prefetch number of DRAM cells to improve the data rate, efficiency is improved by continuously accessing the contents of the same physical bank in which the DRAM is located.

On the other hand, in an encoding technique in which a reference image such as MPEG is searched by motion vector detection and a predicted image is generated, random access in units of pixels is required.
For this reason, when the DRAM access granularity (the size of data acquired in a single access) becomes coarser than the random access unit in units of pixels, it is possible to read and write to a larger area corresponding to the DRAM access granularity. Become.

  For example, in H.264, the minimum size of the color difference reference image is 2x2 pixels, whereas the granularity of 16-bit DDR3 is 16 bytes, and even if the arrangement of pixels and DRAM is devised, it is the worst to obtain 4 bytes of data. ~ 64 bytes of data must be accessed.

  In the prior art, since pixels that are read in vain are highly likely to be used in subsequent processing, measures such as reducing physical access using a buffer memory or a cache memory are taken.

  However, in principle, the macroblock processing proceeds in a zigzag manner in the MPEG processing order, that is, in the raster scan direction. Therefore, in order to increase the data reuse efficiency in reading, a certain amount of data is needed when the data is folded horizontally in the raster scan order. A memory capacity equivalent to a line memory with a large width (or “can hold some data at the top”) is required, but in a low-cost system, having a large cache memory increases the cost of LSI. It becomes a factor. On the other hand, if the hit rate of the cache memory is low, the memory bandwidth that is many times the bandwidth required for actual effective data access is required, which increases the cost and power of the entire system including LSI and DRAM.

  The subject of this invention is providing the motion vector detection apparatus which can reduce the cost and electric power of a circuit.

  A motion vector detection device according to one aspect of the present invention is a motion vector detection device that detects a motion vector when predictive encoding image data with reference to a reference image, and a luminance of a part of the image data in an external memory A luminance pixel storage unit for storing pixel data; a color difference pixel storage unit for storing a part of the color difference pixel data of the image data in an external memory; and the image data and the reference image data based on the luminance pixel data; The memory access cost is set based on whether or not the color difference image data of the image data for which the matching cost has been calculated is already stored in the color difference pixel storage unit. Based on the total cost obtained by adding the matching cost and the memory cost. And a motion vector detecting unit that detects a torque.

  ADVANTAGE OF THE INVENTION According to this invention, the motion vector detection apparatus which can reduce the cost and electric power of a circuit can be provided.

It is a figure which shows the structure of the motion vector detection apparatus of embodiment of this invention. It is a block diagram of the configuration of the access penalty evaluation circuit of the present embodiment. It is a processing flow of the motion vector detection process of this embodiment. It is a figure explaining access cost. It is a figure which illustrates the access cost calculation method of FIG. FIG. 3 is a diagram (part 1) illustrating an image of cost when a vector value is an integer and a decimal. FIG. 6 is a diagram (part 2) illustrating an image of cost when a vector value is an integer and a decimal. It is a figure illustrated about the case where a partition is other than 2x2 pixel. It is the figure which represented the mode of cost as a curved surface.

  In the present embodiment, the cost and power of LSI and DRAM are reduced by adding the memory access cost at the time of motion vector detection so that the memory access can be reduced even in a small-sized buffer / cache memory.

FIG. 1 is a diagram showing a configuration of a motion vector detection device according to an embodiment of the present invention.
The image data stored in the external memory 17 is stored in the search memory (luminance pixel buffer) 10 and the color difference pixel buffer (cache) 11 via the memory access control unit 16. The search memory 10 is a buffer for storing luminance pixel data, and the chrominance pixel buffer 11 is a buffer for storing chrominance pixel data. The luminance pixel data read from the search memory 10 is output as a reference image consisting only of luminance pixels via a sub-pixel filter 12 such as an FIR filter. The decimal pixel filter 12 converts a region indicated by a motion vector having a vector value including a decimal number into a pixel region indicated by an integer value. The luminance pixel data read from the search memory 10 is input to the luminance motion vector detection circuit 14. The luminance motion vector detection circuit 14 detects a motion vector from the luminance pixel data and outputs a motion vector as a detection result. A command for reading the color difference data of the area of the reference image indicated by the motion vector is given to the external memory 17 via the memory access control unit 16. From the external memory 17, the color difference pixel data of the reference image in the area indicated by the motion vector is loaded into the color difference pixel buffer 11. Also, the color difference pixel data read from the external memory 17 is output as a color difference reference image after being filtered by a sub-pixel filter 13 such as FIR.

  The access penalty evaluation circuit 15 obtains information regarding the (color difference) motion vector detected by the luminance motion vector detection circuit 14 and the size of the reference image area indicated by the motion vector. The access penalty evaluation circuit 15 obtains information about whether or not the area indicated by the motion vector is stored in the color difference pixel buffer 11 from the color difference pixel buffer 11. The access penalty evaluation circuit 15 detects the luminance motion vector by using a small value as the access cost when the area indicated by the motion vector is already stored in the color difference pixel buffer 11 and when the area is not stored. Input to the circuit 14.

  The luminance motion vector detection circuit 14 calculates the sum of absolute difference (SAD) or sum of absolute transformed difference (SATD) between the pixel area to be processed and the reference pixel area so that these are minimized. Detect motion vectors. At this time, in the present embodiment, the coefficient value given from the access penalty evaluation circuit 15 as the access cost is added to SAD or SATD. Then, a motion vector is detected so as to minimize SAD or SATD to which the access cost is added.

  The low access cost value is, for example, 0, and the high value is appropriately determined in relation to the average values of SAD and SATD. By adjusting the access cost as a parameter, it is possible to adjust how much the access cost to the memory is taken into account when detecting the motion vector. The larger the value of the access cost is, the more the access cost is considered, and the smaller the value is, the smaller the proportion of the access cost considered in the motion vector detection. The actual value of the access cost should be determined appropriately by the designer of the motion vector detection apparatus using this embodiment.

As described above, the memory access penalty for generating the color difference motion compensated image at the time of luminance search is added to the cost (SAD, SATD) when detecting the motion vector.
In a memory that increases efficiency by burst access such as DDR, an access exceeding a required pixel occurs depending on the granularity of the memory access. When a color difference data access is performed for a certain motion vector, an actual memory access is estimated and reflected as a cost.

  The data present in the color difference pixel buffer (cache) has no penalty. When a new memory access is performed, the access cost is added to the motion vector detection cost based on the valid data, the data to be read wastefully, the coordinates, and the size.

FIG. 2 is a block diagram showing the configuration of the access penalty evaluation circuit according to this embodiment.
When the color difference motion vector and the region size of the search candidate region are input, the region expansion / division unit 22 should read which block to read from the cache the region indicated by the color difference motion vector and the region size. To decide. In other words, the cache stores image data in units of blocks, and in order to read out pixel data of the area, a plurality of blocks partially including the area may need to be read. In that case, it is necessary to read an area that is one size larger than the area, but what kind of area is this one-larger area, and what block should be read from the cache to read this large area To decide. As a result, access alignment information, which is a method for accessing the cache, is generated and passed to the access cost calculation / accumulation circuit 21. The access cost calculation / accumulation circuit 21 also receives the color difference motion vector and its region size. Based on these pieces of information, the access cost calculation / accumulation circuit 21 determines which block is to be accessed, and inquires of the cache (color difference pixel buffer) 11 whether or not the block is in the cache. In response to this inquiry, the cache 11 may or may not have a necessary block, and when it exists, the access cost value is read from the coefficient table 23 and output as the access cost. The control unit 20 is connected to the host CPU interface, and initializes the area expansion / division unit 22 and the access cost calculation / accumulation circuit 21 or updates the coefficient in the coefficient table 23.

FIG. 3 is a process flow of the motion vector detection process of the present embodiment.
First, in step S10, the motion vector detection device is initialized and the search memory is updated. In step S11, a block matching cost is calculated. This calculates a matching cost (total cost of SAD, SATD, and access cost) between the processing target block and the reference block. In step S12, it is determined whether all combinations of search ranges and partitions have been processed. If the determination in step S12 is No, the process returns to step S11. If the determination is Yes, the process proceeds to step S13. In step S12, when the cost is calculated by sequentially moving the search range within the reference image, it is determined whether or not the search range has been moved throughout the reference image, and the size of the block to be matched is determined. It is determined whether matching has been performed for all variations. The block size for matching is basically a 16 × 16 pixel macroblock. However, H. In H.264, etc., processing can be performed in units of blocks in which the macroblocks are further finely divided. Therefore, matching is also performed on these small macroblocks, and the block having the best matching cost is adopted. This division of blocks is called a partition. In the case of MPEG-2, the processing unit is only a macro block of 16 × 16 pixels, so there is only one partition. In step S13, a vector having the minimum cost is set as a color difference motion vector, and the type of partition at that time is determined. In step S14, color difference motion compensation data is acquired, the cache is updated, and the motion vector detection process is terminated.

  Steps S15 to S18 are details of the block matching cost calculation step of step S11. In step S15, the apparatus is initialized, and in step S16, the luminance cost (SAD, SATD) is calculated. In step S17, an access cost is calculated based on the color difference pixel data. In step S18, the values of steps S16 and S17 are added together. As shown here, matching costs (SAD, SATD) are calculated using only luminance pixel data, while access costs are calculated using only chrominance pixel data.

  Steps S19 to S25 are details of the access cost calculation process of step S17. In step S19, the device is initialized, and in step S20, the access cost coefficient is updated. Step S20 does not necessarily need to be updated to a new value, but may be fixed to a fixed value. In step S21, the color difference vector and the area size are fetched. In step S22, a block of a cache access unit necessary for reading pixel data in an area indicated by the color difference vector and the area size is determined. In step S23, an access cost is calculated and cumulatively added. This is because the coefficients determined to be 0 for two blocks, the first value for four, the second value for four, etc. are divided when the number of blocks required to access the area is one. Calculate for each area (block) and cumulatively add. In step S24, it is determined whether or not all divided areas (blocks) have been processed. All blocks in step S24 mean all blocks necessary for reading the area indicated by the color difference motion vector and the area size. If the determination in step S24 is No, the process returns to step S23, and if Yes, the process proceeds to step S25. In step S25, the access cost is output.

There are two methods for setting the access cost: a method for determining a value in advance and a method for dynamically setting the access cost as follows.
FIG. 4 is a diagram for explaining the access cost.

  The basic idea is the same regardless of the shape of the boundary of the region specified by the color difference motion vector and the region size. For example, the idea is the same whether it is a raster or rectangular shape. The area is expanded to the area (block that is an access unit of the cache) that is actually accessed from the coordinates (address) and size of the required color difference area, and the access penalty is evaluated.

  In FIG. 4, (1) is a case where two blocks are required to read an area indicated by the color difference motion vector and the area size, and (2) is a case where four blocks are required. As an example of how to give the cost, in the case of (1), the left and left useless read amounts of the area are determined, and the left use cost costL is multiplied by the left useless read amount. And the right cost, costR, can be added to the right amount of wasted read. At this time, since the data on the right side is highly likely to be reused (there is a high possibility of being reused because it is read more), it is possible to satisfy costL <= costR. In the case of (2), it is possible to calculate the wasteful reading amount on the right side and the wasteful reading amount on the left side and calculate the cost in the same way. However, in this case, the blocks are divided into upper and lower parts. Further, the cost may be increased by adding a fixed value.

  The lower diagram of FIG. 4 is an image of costs. If the new read is biased to the right, the memory access area is the same, but valid data is biased to the left. On the other hand, if the new read is biased to the left, the valid data is biased to the right. When the access cost is calculated as in (1), the memory access cost is smaller when the new read is biased to the right, and the final cost is the new read when the motion vector detection cost (SAD, SATD) is the same. Is smaller when is biased to the right.

  As described above, in addition to preparing the access cost as a fixed value depending on the number of access blocks in the cache, it is also possible to set the access cost to change dynamically according to the actual amount of wasted read. It is.

FIG. 5 is a diagram illustrating the access cost calculation method of FIG.
Pixels finally required for the reference image of the 2 × 2 pixel data of the color difference data vary depending on the fractional component of the vector value of the color difference motion vector. When applying a filter, the size of the filter is made to match the integer value of the vector value, so if there is a fractional component in the vector value, it is necessary to take up one pixel outside. The above-described cost is output based on the color difference motion vector corresponding to the luminance motion vector search and the partition shape. In the example of FIG. 5, the case of 1 accesses 4 alignments (blocks of cache access units), the case of 2 has 4 (1 if the vector value is an integer), and the case of 3 has 2 (vector) If the value is an integer, it is 1). Then, as shown in FIG. 4, the cost is calculated and added based on each amount of useless reading data.

  In FIG. 5, when the vector value includes a decimal, a pixel far from the center-of-gravity pixel is illustrated in gray hatching as a non-white. Comparing the case where only Norishiro crosses the alignment and the case where the integer pixel crosses the alignment has the same motion detection cost, the operation becomes difficult to select in consideration of the cost when only Norishiro crosses. Data existing in the color difference buffer memory is considered to have no penalty.

6 and 7 are diagrams showing an image of cost when the vector value is an integer and a decimal.
FIG. 6 shows an example of the motion vector detection cost before and after the boundary and the memory access cost, and shows a case where costL and costR in FIG. 4 are equal. As the vector value changes from 0 to 4 + decimal, the valid data gradually moves from the left to the right of the boundary. The memory access area is 2 blocks (alignment) in the case of 0 + decimal to 3 + decimal. The memory access cost increases accordingly from 0 + decimal to 3 + decimal. Assuming that the motion detection cost changes as shown in FIG. 6, the case where the vector value is 4 is the smallest in the final cost. Therefore, 4 is adopted as the motion vector value.

  FIG. 7 shows an example in which some of the data to be read in the motion vector detection cost and the memory access cost before and after the boundary are in the cache memory. In the case of FIG. 7, when the vector value is 0, the valid data is read in the past and exists in the buffer, and the memory access cost is 0. When the motion detection cost is as shown in FIG. 7, the final cost is minimum when the vector value is zero. Therefore, 0 is adopted as the motion vector value.

FIG. 8 is a diagram illustrating a case where the partition is other than 2 × 2 pixels.
Similarly, for other shapes such as 4 × 4, 8 × 8, 8 × 4, etc., the vector value is an integer and the white portion of FIG. 8 is included. It is possible to return an evaluation value. In FIG. 8, in the case of 5 and 6, the vector value includes a decimal value, and it is necessary to read up to the gray portion outside the 4 × 4 pixel region, and the alignment is straddled. 7 and 8 are cases where the vector value is an integer, and the block to be read may be 4 × 4, but the block itself straddles the alignment.

  Shapes such as 4 × 4, 8 × 8, and 8 × 4 are examples of color differences in H.264. However, considering the number of taps of the filter in the case of other standards as well, if the vector value is an integer, there is no noise, but if a decimal value is included, a noise is required to the outside of the block. It is possible to think like this.

FIG. 9 is a diagram showing the state of cost as a curved surface.
FIG. 9 shows an example in which the cost calculated by the access penalty detection circuit is taken into account depending on the coordinates and the accessed size. 9A shows a case where there is no access cost, FIG. 9B shows a case where the reference image area indicated by the motion vector and the area size is small, and FIG. 9C shows a case where the reference image area is large. . When there is no access cost, the optimal motion vector is selected as the lowest position on the curved surface in FIG. On the other hand, considering the access cost, as shown in FIGS. 9B and 9C, a step is formed on the curved surface of the cost, and the lowest position of the curved surface is a portion avoiding the step. In the case of FIG. 9C, the reference image area is large, and in this case, the width of the step is wider than that in FIG. In these figures, when there is data read out previously in the color difference pixel buffer, no penalty is set (because it is not read out from SDRAM).

  As described above, the corresponding color difference access cost is calculated in accordance with the motion detection calculation using the luminance data. At this time, a part of the data accessed in the past is held in the color difference buffer, and for data existing there, the cost over the alignment is not added or the access cost is set to 0 (to the external memory). Does not occur, and the buffer data is used).

  The amount of valid / invalid data in the new memory access area is calculated, and the cost is calculated by assigning another weight to the area including the left and right edges of the effective area. When motion detection for 1 MB (macroblock) is completed, the color difference pixel data is read out, the update is instructed to the holding buffer, and it prepares for motion detection of the next macroblock

Note that the cache memory has the following functions.
・ Function to read out insufficient data from external RAM ・ Retention of newly read data for color difference motion compensation ・ Easy implementation to retain data for the past several times in a FIFO or less frequently used Provides an algorithm to discard from data ・ Hold data output ・ Hold data inquiry ・ Function to respond to an inquiry from the access evaluation circuit for data held in the buffer

  In the above embodiment, the value of the access cost is changed when the reference image area straddles the cache alignment. However, when a scene change is confirmed with the reference image, the access cost is temporarily changed. The coefficient for calculating the value may be changed (increased). By doing in this way, when the processing target image and the reference image are completely different due to the scene change, it is possible to refer only between images having much higher similarity than normal inter-frame prediction. In other words, when the scene changes, the similarity between images decreases, so the accuracy of inter-frame prediction falls. Therefore, in order to maintain the accuracy, the case where the similarity is very high is extracted. is there.

In addition to the above embodiment, the following supplementary notes are disclosed.
(Appendix 1)
In a motion vector detection device for detecting a motion vector when predictive encoding image data with reference to a reference image,
A luminance pixel storage unit that stores luminance pixel data of a part of the image data in an external memory;
A color difference pixel storage unit for storing a part of the color difference pixel data of the image data in an external memory;
Based on the luminance pixel data, the matching cost between the image data and the reference image data is calculated, and the color difference image data of the image data that is the calculation target of the matching cost is already stored in the color difference pixel storage unit. A cost setting unit that sets a memory access cost based on whether or not, and adds this to the matching cost;
A motion vector detection unit that detects a motion vector based on the total cost obtained by adding the matching cost and the memory cost;
A motion vector detection device comprising:
(Appendix 2)
The cost setting unit sets the memory cost to 0 when the image data subjected to the calculation of the matching cost is already stored in the color difference pixel storage unit. Motion vector detection device.
(Appendix 3)
The motion vector detection device according to appendix 1, wherein the chrominance pixel storage unit stores the chrominance pixel data in blocks of access units.
(Appendix 4)
The movement according to appendix 3, wherein the cost setting unit sets the memory cost to a predetermined value when the image data subjected to the calculation of the matching cost spans a plurality of the blocks. Vector detection device.
(Appendix 5)
The motion vector detection device according to appendix 4, wherein the predetermined value of the memory cost is set to be proportional to the amount of data read from the external memory.
(Appendix 6)
The cost setting unit adds the cost value corresponding to the scene change to the matching cost when there is a scene change between the processing target image and the reference image. Motion vector detection device.
(Appendix 7)
2. The motion vector detection device according to appendix 1, wherein the matching between the image data and the reference image data is performed in a macro block unit of 16 × 16 pixels or a block unit obtained by further finely dividing the macro block. .
(Appendix 8)
A luminance pixel storage unit for storing luminance pixel data of a part of the image data in the external memory for detecting a motion vector when the image data is predictively encoded with reference to a reference image, and the image in the external memory In a control method in a motion vector detection device comprising a chrominance pixel storage unit that stores chrominance pixel data of a part of data,
Based on the luminance pixel data, a matching cost between the image data and reference image data is calculated,
Based on whether or not the color difference image data of the image data for which the matching cost is to be calculated is already stored in the color difference pixel storage unit, a memory access cost is set,
Adding the memory access cost to the matching cost;
Detecting a motion vector based on the total cost obtained by adding the matching cost and the memory cost;
A control method characterized by that.

10 Search memory (Luminance pixel buffer)
11 Color difference pixel buffer (cache)
12, 13 Decimal pixel filter 14 Luminance motion vector detection circuit 15 Access penalty evaluation circuit 16 Memory access control unit 17 External memory 20 Control unit 21 Access cost calculation / accumulation circuit 22 Area expansion / division unit 23 Coefficient table

Claims (5)

In a motion vector detection device for detecting a motion vector when predictive encoding image data with reference to a reference image,
A luminance pixel storage unit that stores luminance pixel data of a part of the image data in an external memory;
A color difference pixel storage unit for storing a part of the color difference pixel data of the image data in an external memory;
Based on the luminance pixel data, the matching cost between the image data and the reference image data is calculated, and the color difference image data of the image data that is the calculation target of the matching cost is already stored in the color difference pixel storage unit. A cost setting unit that sets a memory access cost based on whether or not, and adds this to the matching cost;
A motion vector detection unit that detects a motion vector based on the total cost obtained by adding the matching cost and the memory cost;
A motion vector detection device comprising:   2. The cost setting unit according to claim 1, wherein the cost setting unit sets the memory cost to 0 when the image data subjected to the calculation of the matching cost is already stored in the color difference pixel storage unit. The motion vector detection device described.   The motion vector detection device according to claim 1, wherein the chrominance pixel storage unit stores the chrominance pixel data in blocks of access units.   The said cost setting part sets the said memory cost to predetermined value, when the said image data used as the calculation object of the said matching cost spans the said several block, The said memory cost is set to predetermined value. Motion vector detection device.   5. The motion vector detection device according to claim 4, wherein the predetermined value of the memory cost is set to be proportional to the amount of data read from the external memory.