JP2007088910A - Motion vector detecting device and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a motion vector detecting device that is small in circuit scale, and to provide an imaging apparatus that utilizes such a motion vector detecting device. <P>SOLUTION: A unit block of a criterion block, stored in a criterion block data storage SRAM 2a, is inputted to correlation operation sections 3a-3h in common, and a unit block of a reference block, resulting from horizontally shifting the reference block pixel by pixel in a unit block of a search area stored in a search area data storage SRAM 2b, is inputted to the correlation operation sections 3a-3h, respectively. Each time the unit block of the criterion block and the unit block of the reference block are inputted, the correlation operation sections 3a-3h perform correlation operations. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a motion vector detection device and an imaging device.

  Motion vectors are used in the fields of motion image stabilization and motion image compression. As a motion vector detection method, a method using a block matching method is widely known. The block matching method is a method for detecting the amount of motion between two images (one of which is called a standard frame and the other is called a reference frame).

  Hereinafter, a simple principle of the block matching method will be described with reference to the drawings. In the block matching method, a correlation operation is performed between a specific area (a reference block) in a reference frame and a reference block in a reference frame having the same block size as the reference block (generally, an absolute difference sum calculation or a difference). The motion vector is obtained by detecting the position of the reference block having the highest correlation with the base block.

  When performing the correlation calculation, a reference block 301b is set in the reference frame 300b as shown in FIG. 17B with respect to one reference block 301a in the reference frame 300a as shown in FIG. To do. Then, correlation calculation is performed between the reference block and the reference block while shifting the position of the reference block 301b by one pixel in the horizontal and vertical directions within a certain search area 302b.

  FIG. 18 is a diagram illustrating a configuration of a conventional motion vector detection device. Here, FIG. 18 shows a configuration when the block sizes of the base block and the reference block are 4 pixels × 4 lines.

  18 includes a main memory 401, a reference frame register group 402a, a reference frame register group 402b, a correlation calculation unit 403, a minimum value detection unit 404, and a timing circuit 405. .

  The main memory 401 is composed of, for example, SDRAM, and stores the above-described reference frame data (reference frame data) and reference frame data (reference frame data). The reference frame register group 402a includes a plurality of registers corresponding to the block size of the reference block (in the example of FIG. 18, the registers are configured by flip-flops (FF)). The stored reference frame data is acquired and held in units of reference blocks. The reference frame register group 402b includes a plurality of registers (FF) corresponding to the block size of the reference block and a plurality of delay devices that delay data by an amount corresponding to the area size of the search area. Reference frame data is acquired and stored in units of search areas.

  The correlation calculation unit 403 acquires the data of the standard block output from the standard frame register group 402a and the data of the reference block output from the reference frame register group 402b, and performs correlation calculation between the standard block and the reference block. Do. Note that the correlation calculation unit 403 in FIG. 18 shows a circuit that calculates the sum of absolute differences. In this case, the correlation calculation unit 403 calculates the difference absolute value for each pixel of the base block and the reference block, The total sum thereof is obtained and output to the minimum value detection unit 404.

  The minimum value detection unit 404 obtains a motion vector based on the position of the reference block when the correlation calculation result in the correlation calculation unit 403 is the smallest. The minimum value detection unit 404 includes a comparator 404b, a flip-flop (FF) 404c, a selector (SEL) 404d, and an FF 404e. The comparator 404b compares the correlation calculation result in the correlation calculation unit 403 with the previous correlation calculation result. The FF 404c holds the output of the comparator 404b as the previous correlation calculation result. In response to the output from the comparator 404b, the SEL 404d outputs either the current reference block position information or the previous reference block position information. The FF 404e holds the output of the SEL 404d as the position information of the previous reference block.

  The timing circuit 405 outputs a signal indicating the position information of the reference block to the minimum value detection unit 404 every time the reference block is moved by one pixel.

  In such a configuration, the reference frame data in the main memory 401 is read and serially converted by the reference frame register group 402a in units of reference blocks, and the data of all the pixels in the reference block are correlated at one time. Is output. Also, the reference frame data in the main memory 401 is read out by the reference frame register group 402b in search area units and subjected to serial-parallel conversion, and the data of all the pixels in the reference block are output to the correlation calculation unit 403 at a time. Here, data outside the reference block that is not used for the current correlation calculation among the data read in units of search areas is held in the delay unit.

  The correlation calculation unit 403 acquires the data of all pixels of the reference block output from the reference frame register group 402a and the data of all pixels of the reference block output from the reference frame register group 402b, and calculates the absolute difference value of each pixel. Are calculated, and the sum of them is obtained and output to the minimum value detection unit 404.

The comparator 404b in the minimum value detection unit 404 compares the correlation calculation result in the correlation calculation unit 403 with the previous correlation calculation result held in the FF 404c, and outputs the smaller one to the SEL 404d. The SEL 404d outputs either the reference block position information input from the current timing circuit 405 or the previous reference block position information held in the FF 404e according to the output of the comparator 404b. Thereby, when the correlation calculation is completed, the position information of the reference block that is finally output from the minimum value detection unit 404 becomes a motion vector.
Such a motion vector detection circuit is proposed in Patent Document 1, for example.
Japanese Patent Laid-Open No. 10-255047

  Here, although the block matching method can detect an accurate motion vector, there is a problem that the circuit scale tends to be large. For example, the motion vector detection apparatus as proposed in Patent Document 1 increases the number of registers constituting the base frame register group and the reference frame register group as the block sizes of the base block and the reference block increase. Therefore, the circuit scale tends to be large.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide a motion vector detection device with a small circuit scale and an imaging device using such a motion vector detection device.

  In order to achieve the above object, a motion vector detection device according to the first aspect of the present invention divides a reference frame into a plurality of reference blocks, sets a search area corresponding to the reference frame in a reference frame, and In a motion vector detection device that detects a motion vector by performing a correlation operation between a reference block and a reference block in the search area, a reference block data storage unit that stores data of the reference block, A search area data storage unit for storing data, and a plurality of correlation calculation units for performing a plurality of different correlation calculations using the data of the reference block and the data of reference blocks in the data of the search area, A reference block obtained by dividing the reference block data by a predetermined number of lines in common to a plurality of correlation calculation units. The reference block data is input in the horizontal direction in a unit block of search area data obtained by dividing the search area data into the predetermined lines for each of the plurality of correlation calculation units. Each of the data of unit blocks of a plurality of reference blocks obtained by shifting is input.

  In this first aspect, a plurality of reference blocks are obtained by commonly inputting unit blocks of the base block to a plurality of correlation calculation units and shifting the reference blocks by one pixel in the horizontal direction in the unit blocks of data in the search area. By inputting each unit block, most of the inputs to the plurality of correlation calculation units can be made common. Thereby, the circuit scale can be reduced.

  In order to achieve the above object, an imaging apparatus according to a second aspect of the present invention includes an imaging unit that captures a subject a plurality of times to obtain reference frame data and reference frame data, and the imaging unit The motion vector detection device according to the first aspect, wherein the motion vector is detected by performing the correlation calculation from the reference frame data and the reference frame data, and the motion vector detected by the motion vector detection device And an image processing unit that performs at least one of a camera shake correction process and a moving image compression process.

  According to the second aspect, the motion vector detection device of the first aspect can be used for the camera shake correction process and the moving image compression process in the imaging apparatus.

  According to the present invention, it is possible to provide a motion vector detection device having a small circuit scale and an imaging device using such a motion vector detection device.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing a configuration of a motion vector detection device according to an embodiment of the present invention. Here, in the configuration of FIG. 1, the block size of the reference block is 16 pixels × 16 lines as shown in FIG. 2A, and the area size of the search area is 23 pixels × 23 lines as shown in FIG. 2B. This is an example. Here, the block size of the reference block is the same as the block size of the base block. Therefore, if the center position of the search area is the reference position of the reference block, the reference block moves within the search area in a range of −4 pixels to +3 pixels in the horizontal direction and −4 pixels to +3 pixels in the vertical direction.

  1 includes a main memory 1, a reference block data storage SRAM 2a, a search area data storage SRAM 2b, correlation calculation units 3a to 3h, a minimum value detection unit 4, a timing circuit 5, and statistical processing. Part 6.

  The main memory 1 is composed of, for example, SDRAM, and stores reference frame data (reference frame data) and reference frame data (reference frame data). Here, the reference frame is divided into a plurality of reference blocks. Also, a plurality of search areas corresponding to the respective reference blocks are set in the reference frame.

  The reference block data storage SRAM 2a is an SRAM having a capacity sufficient to store at least one reference block data (reference block data). The reference block data stored in the main memory 1 is used as a reference block data. Get for each unit block of the block. Here, the unit block of the reference block means unit block data obtained by dividing the reference block data into predetermined lines. In the following examples, the description will be continued assuming that the predetermined line is one line. In this case, the reference block data is divided into unit blocks of 16 reference blocks, and the unit block of each reference block is composed of 16 pixel data.

  FIG. 3A shows a configuration of the reference block data storage SRAM 2a. The example in FIG. 3A is a case where the data amount of one pixel is 8 bits. As shown in FIG. 3A, the reference block data storage SRAM 2a is configured to have a capacity of 16 Word × 128 bits so that 16 unit blocks of the reference block can be stored. FIG. 3B shows data stored at each address of the reference block data storage SRAM 2a. As shown in FIG. 3B, a unit block of one reference block is stored in each address of the reference block data storage SRAM 2a. That is, when the unit block data of the reference block is set as described above, one address of the reference block data storage SRAM 2a has the same vertical position data (16 pixels) in the reference block shown in FIG. Is stored.

  The search area data storage SRAM 2b acquires search area data (search area data) from the reference frame data stored in the main memory 1 for each unit block of the search area. Here, the unit block of the search area means unit block data obtained by dividing the search area data into the predetermined lines. In the case of the above-described example, the search area data is divided into 23 search area unit blocks, and each search area unit block is composed of 23 pixel data.

  FIG. 4A is a diagram showing a configuration of the search area data storage SRAM 2b. As shown in FIG. 4A, the search area data storage SRAM 2b is configured to have a capacity of 23 Word × 184 bits so that 23 unit blocks of the search area can be stored. FIG. 4B shows data stored at each address of the search area data storage SRAM 2b. When the unit block of the search area is set as described above, the same vertical position data (23 pixels) in the search area shown in FIG. 2B is stored in one address of the search area data storage SRAM 2b. .

  FIG. 5 is a diagram for explaining the connection between the reference block data storage SRAM 2a and the search area data storage SRAM 2b and the correlation calculation units 3a to 3h. The reference block data storage SRAM 2a is provided with 16 output terminals a0 to a15 for outputting the data of the unit block of the reference block for each pixel, and these output terminals a0 to a15 are provided with the correlation calculation units 3a to 3h. Are commonly connected to the input terminals a0 to a15.

  With such a configuration, the unit block data of one reference block is output from the reference block data storage SRAM 2a every clock, and the unit block data of this reference block is shared by the correlation calculation units 3a to 3h. Entered.

  Further, the search area data storage SRAM 2b is provided with 23 output terminals b0 to b22 for outputting the data of the unit block of the search area for each pixel, and among these output terminals b0 to b22, the output terminal b0 to b15 are connected to the input terminals b0 to b15 of the correlation calculation unit 3a, the output terminals b1 to b16 are connected to the input terminals b1 to b16 of the correlation calculation unit 3b, and the output terminals b2 to b17 are input to the correlation calculation unit 3c. The terminals b2 to b17 are connected, the output terminals b3 to b18 are connected to the input terminals b3 to b18 of the correlation calculator 3d, the output terminals b4 to b19 are connected to the input terminals b4 to b19 of the correlation calculator 3e, and the output terminals b5 to b20 are connected to the input terminals b5 to b20 of the correlation calculation unit 3f, and the output terminals b6 to b21 are connected to the input terminals b6 to b21 of the correlation calculation unit 3g. Is, output terminal b7~b22 is connected to the input terminal b7~b22 correlation calculation section 3h.

  With this configuration, the search area data storage SRAM 2b outputs the data of one search area unit block for each clock, and the 0th to 15th pixel data in the search area unit block. Is input to the correlation calculation unit 3a, the data of the 1st to 16th pixels is input to the correlation calculation unit 3b, the data of the 2nd to 17th pixels is input to the correlation calculation unit 3c, and the data of the 3rd to 18th pixels is correlated. The 4th to 19th pixel data is input to the correlation calculation unit 3e, the 5th to 20th pixel data is input to the correlation calculation unit 3f, and the 6th to 21st pixel data is input to the correlation calculation unit 3d. 3g, and the data of the 7th to 22nd pixels are input to the correlation calculation unit 3h.

  That is, the unit block data of the search area input to each correlation calculation unit corresponds to the data obtained when the reference block is shifted in the range of -4 to +3 in the horizontal direction in the unit block of the search area. To do. Hereinafter, the 16-pixel data input to each correlation calculation unit as described above is referred to as a unit block of the reference block.

  The correlation calculation units 3a to 3h acquire the unit block of the reference block output from the reference block data storage SRAM 2a and the unit block of the reference block output from the search area data storage SRAM 2b, and obtain the unit block of the reference block and the reference block The correlation calculation is performed between the base block data and the reference block data by calculating the correlation between the unit block and obtaining the sum of the correlation calculation results for each unit block.

  FIG. 6 is a diagram illustrating the configuration of the correlation calculation unit. Although FIG. 6 shows only the configuration of the correlation calculation unit 3a, the other correlation calculation units also differ in that the data of the unit block of the reference block shifted by one pixel in the horizontal direction is input. The configuration is the same as that of the correlation calculation unit 3a. FIG. 6 shows the configuration in the case where the difference absolute value sum calculation is used for the correlation calculation, but the difference square sum calculation may be used for the correlation calculation.

  As shown in FIG. 6, the correlation calculation unit 3 a includes a difference absolute value sum calculation unit 31, an adder 32, and a flip-flop (FF) 33.

The difference absolute value sum calculation unit 31 includes the same number (16 in this case) of difference absolute value calculation units as the number of pixels of the unit block input from the reference block data storage SRAM 2a and the search area data storage SRAM 2b, and these difference absolute value calculations. An adder that adds the outputs of the units is calculated, and after calculating the absolute value of the difference between the unit block of the base block and the unit block of the reference block for each pixel, the sum of these is obtained. For example, when the unit block of the base block of the first line and the unit block of the reference block of the first line are input to the correlation calculation unit 3a,
｜ A (0,0) -B (0,0) | + | A (0,1) -B (0,1) | +… + | A (0,15) -B (0,15) | Formula 1)
Is calculated. In addition, when using the difference square sum calculation for the correlation calculation, it is only necessary to perform a calculation for calculating the sum of these by squaring the difference for each pixel instead of the above calculation.

  The adder 32 adds the difference absolute value sum calculation result obtained by the difference absolute value sum calculation unit 31 to the previous difference absolute value sum calculation result. The FF 33 holds the calculation result of the adder 32 as the previous calculation result of the sum of absolute differences.

  With such a configuration, the correlation calculation unit 3a performs the correlation calculation between the unit block of the base block and the unit block of the reference block once every clock, and adds the correlation calculation results. The correlation calculation between the standard block and the reference block is finished with the clock.

  The minimum value detection unit 4 outputs the position of the reference block when the correlation calculation result in the search area is the smallest as a motion vector. The minimum value detecting unit 4 includes a comparator 4a, a comparator 4b, a flip-flop (FF) 4c, a selector (SEL) 4d, and an FF 4e.

  The comparator 4a compares the correlation calculation results in the correlation calculation units 3a to 3h and outputs the smallest one. The comparator 4b compares the correlation calculation result output from the comparator 4a with the previous correlation calculation result and outputs the smaller one. The FF 4c holds the correlation calculation result output from the comparator 4a as the previous correlation calculation result. The SEL 4d outputs either the current reference block position information or the previous reference block position information to the statistical processing unit 6 in accordance with the output from the comparator 4b. The FF 4e holds the position information of the reference block selected by the SEL 4d as the position information of the previous reference block.

  The timing circuit 5 outputs a signal indicating the position information of the reference block at that time to the minimum value detection unit 4 every clock. When a plurality of reference blocks are set in one reference frame, the statistical processing unit 6 performs a statistical process on a plurality of motion vectors obtained for each of the reference blocks, thereby moving the entire screen. A vector is obtained (see FIG. 7).

  In the following, block matching (particularly, correlation operation in block matching) in the motion vector detection device of FIG. 1 will be described in more detail. FIG. 8 is a timing chart relating to block matching of the motion vector detection device. FIG. 9 is a diagram showing an outline of block matching according to the timing chart of FIG. Here, the example of FIG. 9 shows a case where the correlation calculation is performed from the upper left end (horizontal position −4, vertical position −4 of the reference block) in the search area.

  In the example shown in FIG. 8, the correlation calculation results between the reference block and all reference blocks adjacent in the horizontal direction at a certain vertical position can be simultaneously acquired in 16 clocks.

  For example, from the first to the 16th clock, it is possible to obtain correlation calculation results at positions where the vertical direction is −4 and the horizontal direction is −4 to +3. That is, for each clock, from the reference block data storage SRAM 2a, unit blocks (16 pixels) of the reference block are sequentially read from the head address, and the unit blocks of the read reference block are correlated operation units 3a to 3h. Are input in common. Correspondingly, from the search area data storage SRAM 2b, the unit block (23 pixels) of the search area is sequentially read from the head address, and the reference block is shifted pixel by pixel in the read unit block of the search area. The unit blocks of the eight reference blocks obtained in this way are input to the correlation calculation units 3a to 3h, respectively.

  When the unit block of the base block and the unit block of the reference block are input to the correlation calculation units 3a to 3h, the correlation calculation is performed in each correlation calculation unit. Thus, for each clock, the correlation calculation unit 3a has horizontal direction -4, the correlation calculation unit 3b has horizontal direction -3, the correlation calculation unit 3c has horizontal direction -2, the correlation calculation unit 3d has horizontal direction -1, and the correlation calculation Unit block and reference block of horizontal block 0 in unit 3e, horizontal direction +1 in correlation calculation unit 3f, horizontal direction +2 in correlation calculation unit 3g, and horizontal direction +3 in correlation calculation unit 3h (all vertical directions are -4) The correlation calculation result is obtained between the reference block and the reference block in the vertical direction −4 and the horizontal direction −4 to +3 (block line 1 in FIG. 9) in 16 clocks. Eight results can be obtained simultaneously.

  In the same manner, correlation calculation is performed with 16 clocks as one cycle. At this time, the unit block (16 pixels) of the reference block is always sequentially read from the head address from the reference block data storage SRAM 2a every clock. On the other hand, from the search area data storage SRAM 2b, the unit blocks (23 pixels) of the search area are sequentially read while the read start address is shifted one by one in the vertical direction every 16 clocks.

  Such processing is performed for the entire range of the search area. As a result, 8 × 8 = 64 correlation calculation results are obtained from block line 1 to block line 8 (vertical direction −4 to +3 of the reference block).

  When the processing for one reference block is completed, the correlation calculation can be performed on the entire photographed data by sequentially repeating the processing of FIGS. 8 and 9 for the next reference block. .

  Next, a modification of the present embodiment will be described. Here, in the modification described below, the description will be continued assuming that the block size of the reference block is 32 pixels × 32 lines and the area size of the search area is 63 pixels × 63 lines. In this case, the reference block moves within the search area in a range from −16 pixels to +15 pixels in the horizontal direction and from −16 pixels to +15 pixels in the vertical direction.

  In the modification described here, the basic configuration is the same as that shown in FIG. 1, but the internal configurations of the reference block data storage SRAM 2a and the search area data storage SRAM 2b are different from those shown in FIGS. .

  FIG. 10A is a diagram showing a configuration of the reference block data storage SRAM 2a in the modification. As shown in FIG. 10A, the reference block data storage SRAM 2a is configured to store 32 unit blocks of the reference block composed of 32 pixels × 1 line. Here, the difference from FIG. 3A is that one unit block is divided into a plurality of parts (here, as an example, divided into two unit blocks every 16 pixels), and these divided unit blocks (reference The block division unit block) is stored in separate areas in the reference block data storage SRAM 2a. For this reason, the reference block data storage SRAM 2a is configured to have a capacity of 64 Word × 128 bits. Such reference block data storage SRAM 2a acquires the reference block data from the reference frame data stored in the main memory 1 for each divided unit block (16 pixels) obtained by dividing the unit block of the reference block into two. The data stored in the reference block data storage SRAM 2a is read for each division unit block of 16 pixels. Thereby, the data of the division unit block of 16 pixels is commonly input to the correlation calculation units 3a to 3h.

  FIG. 10B shows data stored at each address of the reference block data storage SRAM 2a. As shown in FIG. 10B, at each address of the reference block data storage SRAM 2a, data obtained by dividing a unit block of one reference block into two (16 pixels) is the left area and right side of the reference block data storage SRAM 2a, respectively. It is stored separately for each area.

  FIG. 11A is a diagram showing the configuration of the search area data storage SRAM 2b in the modification. As shown in FIG. 11A, the search area data storage SRAM 2b is configured to store 63 unit blocks of the search area composed of 63 pixels × 1 line.

  Here, in the modification, the search area data storage SRAM 2b includes two SRAMs (SRAM_A21a, SRAM_B21b) having a capacity of 192 Word × 64 bits, one SRAM (SRAM_C21c) having a capacity of 128 Word × 64 bits, and these three SRAMs. And a selector (SEL) 22 for selectively outputting data.

  Each of the three SRAMs of the search area data storage SRAM 2b acquires search area data from the reference frame data stored in the main memory 1 in units of 8 pixels. Here, as shown in FIG. 11B, data of 24 pixels can be acquired by three of SRAM_A 21a, SRAM_B 21b, and SRAM_C 21c. However, the correlation calculation units 3a to 3h each have a division unit block of the reference block. It is only necessary to input 16-pixel data corresponding to (16 pixels). Therefore, the SEL 22 selects only 23 pixels (the data of 23 pixels is referred to as a division unit block of the search area) out of the data output by 24 pixels in units of 8 pixels from each SRAM of the search area data storage SRAM 2b. Output. Of the 23 pixel data output here, the 0th to 15th pixel data is input to the correlation calculation unit 3a, the 1st to 16th pixel data is input to the correlation calculation unit 3b, and the 2nd to 17th pixel data is input. The data is input to the correlation calculation unit 3c, the 3rd to 18th pixel data is input to the correlation calculation unit 3d, the 4th to 19th pixel data is input to the correlation calculation unit 3e, and the 5th to 20th pixel data is The data of the 6th to 21st pixels is input to the correlation calculating unit 3g, and the data of the 7th to 22nd pixels is input to the correlation calculating unit 3h.

  FIG. 11C shows data stored in each address of SRAM_A 21a, SRAM_B 21b, and SRAM_C 21c. As shown in FIG. 11C, data in units of 8 pixels is stored in each address of the SRAM_A 21a, SRAM_B 21b, and SRAM_C 21c. Here, data at the same vertical position in the search area is divided and stored in SRAM_A 21a, SRAM_B 21b, and SRAM_C 21c.

Hereinafter, block matching in the modification will be described with reference to FIG.
In FIG. 12, the correlation calculation results at the positions of −16 in the vertical direction and −16 to −9 in the horizontal direction are acquired from the first to the 64th clock.

  First, from the first clock to the 32nd clock, the division unit block (16 pixels) on the left side of the reference block is sequentially read from the reference block data storage SRAM 2a every clock and the read reference block is read. The division unit block is commonly input to the correlation calculation units 3a to 3h. For example, in the case of the first clock, ADD = 0 data in the reference block data storage SRAM 2a, that is, 16 pixels A (0,0) to A (0,15) are read. Correspondingly, data of 8 pixels is read from each of the SRAM_A 21a, SRAM_B 21b, and SRAM_C 21c of the search area data storage SRAM 2b, and the 23rd pixel from the first pixel among the read 24 pixel data. The data up to are selected in SEL22. For example, in the case of the first clock, ADD = 0 data of SRAM_A 21a, SRAM_B 21b, and SRAM 21c, that is, B (0,0) to B (0,7), B (0,8) to B (0,15 ), B (0,9) to B (0,23) are read out. Of these 24 pixel data, only 23 pixel data from B (0,0) to B (0,22) are stored. Selected by SEL22.

  The search unit division unit block (23 pixels) selected by the SEL 22 is divided into eight reference block division unit blocks (16 pixels) obtained by shifting the reference block by one pixel in the horizontal direction. Each is input to 3h. Subsequent correlation calculations are performed in the same manner as in FIG.

  Thus, for each clock, the correlation calculation unit 3a has a horizontal direction of -16, the correlation calculation unit 3b has a horizontal direction of -15, the correlation calculation unit 3c has a horizontal direction of -14, the correlation calculation unit 3d has a horizontal direction of -13, and the correlation calculation. The left side of the reference block in the horizontal direction -12 in the unit 3e, in the horizontal direction -11 in the correlation calculation unit 3f, in the horizontal direction -10 in the correlation calculation unit 3g, and in the horizontal direction -9 in the correlation calculation unit 3h (all the vertical directions are -16) A correlation calculation result is obtained between the division unit block and the division unit block on the left side of the reference block (the left side of the block line 1-1 in FIG. 12).

  From the 33rd clock to the 64th clock, the division unit block on the right side of the reference block is sequentially read from the start address from the reference block data storage SRAM 2a, and the division unit block of the read reference block is the correlation calculation unit. 3a to 3h are input in common. For example, at the 33rd clock, ADD = 32 data in the reference block data storage SRAM 2a, that is, 16 pixels A (0,16) to A (0,31) are read. Correspondingly, data is read out from the SRAM_A 31a, SRAM_B 31b, SRAM_C 31c of the search area data storage SRAM 2b by 8 pixels (for example, in the case of the 33rd clock, B (0, 16) to B (0 , 23), B (0,24) to B (0,31) from SRAM_A 31a, and B (0,32) to B (0,39) are read from SRAM_B 31b). The data up to the pixel is selected in SEL22, and the division unit blocks (16 pixels) of eight reference blocks obtained by shifting the reference block by one pixel in the horizontal direction in the division unit blocks of these selected search areas are correlated. Each is input to the arithmetic units 3a to 3h. Subsequent correlation calculations are performed in the same manner as in FIG.

  Thus, for each clock, the correlation calculation unit 3a has a horizontal direction of -16, the correlation calculation unit 3b has a horizontal direction of -15, the correlation calculation unit 3c has a horizontal direction of -14, the correlation calculation unit 3d has a horizontal direction of -13, and the correlation calculation. The right side of the reference block in the horizontal direction -12 in the unit 3e, the horizontal direction -11 in the correlation calculation unit 3f, the horizontal direction -10 in the correlation calculation unit 3g, and the horizontal direction -9 in the correlation calculation unit 3h (all the vertical directions are -16) The correlation calculation result between the division unit block of the reference block and the division unit block on the right side of the reference block (the right side of the block line 1-1 in FIG. 12) is obtained. The correlation calculation with the unit block of the block is completed, and eight correlation calculation results are obtained simultaneously.

  In the subsequent processing, the division unit block (16 pixels) of the reference block is sequentially read from the left top address from the reference block data storage SRAM 2a every 64 clocks. On the other hand, from the search area data storage SRAM 2b, the division unit block (23 pixels) of the search area is sequentially read while shifting the read position by 8 pixels in the horizontal direction every 64 clocks.

  By performing such processing four times (for 256 clocks), the reference block and the reference block in the vertical direction -16 and the horizontal direction -16 to +15 (block line 1-1 to block line 1-4 in FIG. 12). As a result, 8 × 4 = 32 correlation calculation results are obtained.

  In the same manner, the correlation calculation is performed with 256 clocks as one cycle. At this time, the division block (16 pixels) of the reference block is always sequentially read from the left head address from the reference block data storage SRAM 2a. On the other hand, from the search area data storage SRAM 2b, division unit blocks (23 pixels) of the search area are sequentially read while shifting the read start address one by one in the vertical direction every 256 clocks.

  Such processing is performed for the entire range of the search area. As a result, 32 × 32 = 1024 correlation calculation results are obtained from block line 1 to block line 32 (vertical position of reference block −16 to +15).

  When the processing for one reference block is completed, correlation processing is performed on the entire captured data by sequentially repeating the processing of FIG. 12 for the next reference block, and the motion vector of the entire screen is determined. Can be requested.

  Next, a digital camera as an example of an imaging apparatus to which the motion vector detection apparatus according to the present embodiment is applied will be described with reference to FIG. The digital camera shown in FIG. 13 includes a CCD 101, a preprocessing unit 102, an image bus 103, a main memory 104, a motion detection device 105, a CPU bus 106, a CPU 107, an image processing unit 108, and a video encoder 109. And a TFT display unit 110.

  The CCD 101 is a CCD image sensor, and captures an image signal by imaging a subject (not shown). The preprocessing unit 102 acquires image data by processing the image signal obtained by the CCD 101. The image bus 103 is a transfer path for transferring various data such as imaging data obtained by the preprocessing unit 102 to each unit of the digital camera in FIG. The main memory 104 is composed of, for example, an SDRAM or the like, and stores image data obtained by the preprocessing unit 102 and transferred via the image bus 103. The motion detection device 105 is configured as shown in FIG. 1, and detects a motion vector as described above from two pieces of imaging data (base frame data and reference frame data) stored in the main memory 104. The CPU bus 106 is a transfer path for transferring various data such as motion vectors detected by the motion detection device 105 to the CPU 107. The CPU 107 controls the entire operation of the digital camera. The image processing unit 108 reads out the imaging data stored in the main memory 104 and performs various image processing. The video encoder 109 converts the imaging data processed by the image processing unit into a signal suitable for display. The TFT display unit 110 displays an image based on the signal converted by the video encoder 109.

  Hereinafter, in the digital camera having the configuration as shown in FIG. 13, an operation when the motion detection device 105 is used for camera shake correction will be described with reference to FIGS. 14 and 15.

  First, the CCD 101 captures an image of a subject at every predetermined timing, and outputs an image signal obtained thereby to the preprocessing unit 102. The preprocessing unit 102 performs preprocessing such as A / D conversion, shading correction, and defective pixel correction on the imaging signal input from the CCD 101 to generate imaging data, and the generated imaging data is output to the image bus 103. Is written into the main memory 104 (FIG. 14A).

  The motion detection device 105 reads out shooting data for two frames from the main memory 104, detects a motion vector by the method described above based on the shooting data for these two frames, and uses this motion vector as a CPU bus. It transfers to CPU107 via 106 (FIG.14 (b)).

  The CPU 107 determines the amount of camera shake generated between the two frames of shooting data from the motion vector transferred via the CPU bus 106, and determines the shooting data clipping position necessary to correct this camera shake. The image processing unit 108 is notified (FIG. 15A). Note that when the CPU 107 determines the cutout position, it is determined after separating the movement of the image due to camera shake and the movement of the image due to the user intentionally moving the digital camera.

  Here, the camera shake correction will be briefly described. Electronic camera shake correction is used for moving camera shake correction. As shown in FIG. 16, the electronic image stabilization is performed by acquiring image data 200 having a wider angle of view than the actual display image data in the CCD 101, and in the digital camera from the image data 200 in the image processing unit 108. On the basis of the amount of movement (the amount of camera shake) between the image 201a and the image 201b, shooting data at a clipping position that corrects the movement of the image is read out from the main memory 104 as shooting data for display.

  The image processing unit 108 acquires shooting data for display from the main memory 104 based on the cut-out position notified from the CPU 107, and performs various image processes necessary for display on the acquired shooting data. Then, the image processing unit 108 writes the processed imaging data back to the main memory 104 (FIG. 15B).

  Thereafter, the video encoder 109 reads out the camera-shake-corrected shooting data from the main memory 104 and converts it into a video signal such as an NTSC signal, and displays a moving image on the TFT display unit 110 based on the converted signal. By such an operation, a moving image that has undergone camera shake correction is displayed on the TFT display unit 110 (FIG. 15C).

  Here, FIGS. 14 and 15 describe camera shake correction at the time of moving image display, but camera shake correction at the time of moving image recording is performed in the same manner. Also, when recording a moving image, MPEG is used for compressing the image data. MPEG compression is a technique for compressing a moving image using both compression using discrete cosine transform (DCT) and compression using motion compensation prediction using a motion vector. The motion vector detection device described in the embodiment and modification of the present invention can be used.

  As described above, according to the present embodiment, the reference block and the search area are each divided into a plurality of unit blocks, and the unit of the reference block among the divided unit block of the reference block and the unit block of the search area is divided. The blocks are input in common to the correlation calculation units 3a to 3h, and the unit blocks in the search area are input to the correlation calculation units 3a to 3h so that the unit blocks of a plurality of reference blocks adjacent in the horizontal direction are input to the correlation calculation units 3a to 3h. The correlation calculation results related to the reference block to be obtained can be obtained simultaneously. That is, since the data input to the correlation calculation units 3a to 3h can be made almost common (the standard block data is made completely common and most of the reference block data is made common), the correlation is made fast with a small circuit scale. It is possible to perform calculations.

  Further, by dividing the storage area of one unit block as in the above-described modification, it is not necessary to increase the circuit scale so much even if the block size increases.

  In the above-described embodiments and modifications, the predetermined line when setting one unit block is one line, but this may be any other number of lines. For example, in the above-described embodiment and modification, when the predetermined line is two lines, two lines of data are stored in one address of the reference block data storage SRAM 2a and the search area data storage SRAM 2b, respectively. Correlation calculation is performed in units of data.

  Further, the block size of the reference block and the area size of the search area are not limited to the above example. In addition, when these sizes are changed, it is preferable to increase or decrease the number of correlation calculation units accordingly.

  In the above-described embodiment and modification, data is read in the horizontal direction, but may be read in the vertical direction. In this case, the unit block is set in the vertical direction.

  In the above-described embodiments and modifications, the reference block data and the search area data are stored in the SRAM. However, these can be substituted by a flip-flop (FF) or the like. However, it is preferable to use SRAM from the viewpoint of circuit scale.

  Although the present invention has been described based on the above embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention.

  Further, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, the problem described in the column of the problem to be solved by the invention can be solved, and the effect described in the column of the effect of the invention Can be extracted as an invention.

It is a figure shown about the structure of the motion vector detection apparatus which concerns on one Embodiment of this invention. FIG. 2A is a diagram showing the reference block, and FIG. 2B is a diagram showing the search area. FIG. 3A is a diagram showing a configuration of the reference block data storage SRAM, and FIG. 3B is a diagram showing data stored at each address of the reference block data storage SRAM. FIG. 4A is a diagram showing a configuration of the search area data storage SRAM, and FIG. 4B is a diagram showing data stored at each address of the search area data storage SRAM. It is a figure for demonstrating the connection with reference | standard block data storage SRAM and search area data storage SRAM, and each correlation calculating part. It is a figure shown about the structure of a correlation calculating part. It is a figure for demonstrating a statistics process part. It is a timing chart which concerns on block matching of a motion vector detection apparatus. It is the figure which showed the outline of the block matching according to the timing chart of FIG. FIG. 10A is a diagram illustrating a configuration of a reference block data storage SRAM according to a modification, and FIG. 10B is a diagram illustrating data stored at each address of the reference block data storage SRAM according to the modification. . FIG. 11A is a diagram showing a configuration of a modified search area data storage SRAM, and FIG. 11B is a diagram for explaining reading of data from the modified search area data storage SRAM. FIG. 11C is a diagram showing data stored at each address of the search area data storage SRAM of the modification. It is the figure which showed the outline of the block matching of a modification. It is a figure shown about the structure of the digital camera as an example of the imaging device to which the motion vector detection apparatus which concerns on one Embodiment of this invention is applied. FIG. 14 is a first diagram for explaining a flow of camera shake correction processing in the digital camera of FIG. 13. FIG. 14 is a second diagram for explaining the flow of camera shake correction processing in the digital camera of FIG. 13. It is a figure for demonstrating electronic camera shake correction. It is a figure for demonstrating the concept of block matching. It is a figure shown about the structural example of the conventional motion vector detection apparatus.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Main memory, 2a ... Reference block data storage SRAM, 2b ... Search area data storage SRAM, 3a-3h ... Correlation calculating part, 4 ... Minimum value detection part, 5 ... Timing circuit, 6 ... Statistical processing part, 101 ... CCD DESCRIPTION OF SYMBOLS 102 ... Pre-processing part 103 ... Image bus 104 ... Main memory 105 ... Motion detection apparatus 106 ... CPU bus | bath 107 ... CPU 108 ... Image processing part 109 ... Video encoder 110 ... TFT display part

Claims (7)

A reference frame is divided into a plurality of reference blocks, a search area corresponding to the reference frame is set in a reference frame, and a motion vector is calculated by performing a correlation operation between the reference block and the reference block in the search area. In the motion vector detection device to detect,
A reference block data storage for storing data of the reference block;
A search area data storage unit for storing data of the search area;
A plurality of correlation calculation units that perform a plurality of different correlation calculations using the data of the standard block and the data of the reference block in the data of the search area;
Comprising
The reference block unit block data obtained by dividing the reference block data into predetermined lines for each of the plurality of correlation calculation units is input, and the search area data is input to each of the plurality of correlation calculation units. A motion vector comprising: a plurality of reference block unit block data obtained by shifting the reference block by one pixel in a horizontal direction in a search area data unit block divided into predetermined lines. Detection device. The unit block data of the reference block is sequentially input to the plurality of correlation calculation units while being shifted by one unit block in the vertical direction, and the unit block data of the reference block is vertically input to the plurality of correlation calculation units. Enter while shifting by one unit block,
Each of the plurality of correlation calculation units receives the unit block data of the reference block and the unit block data of the reference block each time the unit block data and the reference block unit block data are input. The correlation calculation between the base block and the reference block is obtained at the same time when the correlation calculation for all the unit blocks of the base block and the reference block is completed. The motion vector detection device according to claim 1.   Each of the plurality of correlation calculation units performs a correlation calculation between the reference block and the reference block each time the vertical position of the reference block in the search area is shifted by one line, thereby The motion vector detection apparatus according to claim 2, wherein a correlation calculation result with the search area is obtained. The reference block data storage unit is divided into a plurality of areas in which data of unit blocks of the reference block is divided and stored,
3. The motion vector detection device according to claim 2, wherein the search area data storage unit is divided into a plurality of areas in which unit block data of the search area is divided and stored. Each of the plurality of correlation calculation units performs the correlation calculation by at least one of a difference absolute value sum calculation and a difference square sum calculation,
A minimum value detection unit that detects a minimum value of a correlation calculation result in the correlation calculation unit and outputs a position of the reference block when the minimum value is detected as the motion vector. 5. The motion vector detection device according to any one of 1 to 4.   6. The motion vector detection device according to claim 1, wherein each of the reference block data storage unit and the search area data storage unit includes an SRAM. An imaging unit that captures a subject multiple times to obtain data of a reference frame and data of a reference frame;
The motion vector detection device according to any one of claims 1 to 6, wherein the motion vector is detected by performing the correlation calculation from the data of the reference frame obtained by the imaging unit and the data of the reference frame.
An image processing unit that performs at least one of a camera shake correction process and a moving image compression process based on a motion vector detected by the motion vector detection device;
An imaging apparatus comprising:

Images