JP5400573B2 - Image processing apparatus and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus and an imaging apparatus that detect a motion vector and parallax information from a video signal, and more particularly to improvement in detection accuracy and detection processing speed of a motion vector and parallax information.

  Conventionally, motion compensation has been widely used as a method of compressing a video signal. This is to detect a motion vector indicating the moving direction and moving amount of an object (element) in a continuous image (video) frame extracted from a video signal, and compresses the image frame based on this motion vector. It is a way to realize the rate.

As typical methods for detecting a motion vector, methods called block matching and stereo matching have been proposed. Below, this block matching process is demonstrated.
First, in a continuous first image frame and second image frame, a reference macroblock consisting of, for example, 16 vertical pixels × 16 horizontal pixels is set (defined) in the first image frame. Further, a search macroblock having the same size as the reference macroblock is set in the pixel block of the second image frame that is spatially at the same position as the set reference macroblock.

Thereafter, the search macroblock is moved up, down, left and right for every one or a plurality of pixels, and the absolute value of the difference between the pixel value of the pixel of the reference macroblock and the pixel value of the search macroblock corresponding to the pixel is calculated, The position of the search macro block that minimizes the sum of the absolute difference values for all the pixels in the block, that is, the absolute difference value sum is specified. Then, this position is regarded as a macroblock to which the reference macroblock is moved, and a motion vector of the reference macroblock is obtained.
As the pixel value, a value corresponding to brightness is used. For example, the sum of red pixels, green pixels, and blue pixels in the vicinity of the target position is used as a brightness value, or a brightness value is calculated using a predetermined determinant. Can be obtained.

  In addition, if the same subject is imaged from imaging devices arranged at different viewpoint positions, the distance from the viewpoint position to the subject can be calculated based on the parallax information. In motion vector detection, block matching is performed on two consecutive image frames. When calculating the distance to a subject, for example, the same subject output by two imaging devices arranged at two different viewpoint positions is used. By performing block matching on the two image frames and detecting parallax information for the same subject, the distance from the imaging device to the subject can be obtained.

  By the way, there is a case where a motion vector is detected based on an image frame output by an imaging device provided with a Bayer array R / G / B filter corresponding to each pixel of the imaging device. This image frame is RAW data (raw data) that is not subjected to demosaic processing.

Various methods for detecting a motion vector at high speed with respect to an object in such an image frame have been proposed.
For example, in Patent Document 1, as shown in FIG. 16, an imaging provided with a Bayer array filter in which R, G, B filters (color pixels) and transparent filters (white pixels) are periodically arrayed is provided. A motion vector detection device that extracts a brightness value of a white pixel (P pixel) from an image frame output from an element, performs block matching based on the brightness value, and detects a motion vector at high speed is disclosed.

JP 2008-141706 A

  However, in the method disclosed in Patent Document 1, if a motion vector is detected by block matching based on a pixel value (luminance value) of a white pixel, there is a possibility that erroneous detection occurs. For example, it is assumed that a red object X and a blue object Y having the same brightness are in the first image frame and the second image frame. Since the brightness of the red object X and the brightness of the blue object Y are the same, there is no difference in the luminance values of the pixels of both the objects X and Y.

Therefore, to calculate the macroblock M _X corresponding to the area of the red object X of the first image frame, a sum of absolute differences between the macroblock M _Y corresponding to the area of the blue object Y of the second image frame When performing matching of both macroblocks Te, misrecognized macroblock M _Y corresponding to the blue object Y of the second image frame is the destination block of the macroblock M _X corresponding to the red object X of the first image frame As a result, the accuracy of motion vector detection is greatly reduced.

  In addition, when transparent pixels are arranged as disclosed in Patent Document 1 in order to detect luminance values with high accuracy, the number of green pixels is reduced, the color resolution is reduced, and the image captured by the imaging device is reduced. The image quality will deteriorate. Furthermore, although the influence of noise on transparent pixels can be reduced, the transmittance of color pixels is only about 1/3 that of transparent pixels, so the amount of light is also reduced to 1/3, and noise is increased, and imaging is performed by an imaging device. The image noise will increase and the image quality will deteriorate.

  On the other hand, it is possible to perform block matching using the pixel values of the red pixel, the green pixel, and the blue pixel as they are. This can reduce the possibility of erroneous detection due to the difference in color, but causes a problem that the amount of calculation increases. In other words, when performing block matching using luminance, matching is performed for one channel of the luminance channel, but using red, green, and blue channels requires matching for three channels, and the calculation amount is at least three times or more. Become.

  The same problem occurs not only in motion vector detection but also in detection of parallax information.

  The present invention has been made in view of such a situation, for example, based on an image frame output by an imaging device provided with a Bayer array R, G, B filter corresponding to each pixel of the imaging device, An object of the present invention is to provide an image processing apparatus that detects a motion vector and parallax information with high accuracy and at high speed when detecting a motion vector and parallax information by a block matching method, and an imaging apparatus including the apparatus.

The first technical means includes a reference macroblock setting unit that sets a reference macroblock in a first image frame having a Bayer array-like primary color pixel arrangement, and a Bayer array-like configuration of the reference macroblock set in the first image frame. A search macroblock setting unit configured to set a search macroblock for the second image frame so that the primary color pixel arrangement of the same Bayer arrangement as the primary color pixel arrangement is obtained, and the search macroblock setting unit includes the reference macroblock The block setting unit sets a search macroblock having the same size as the reference macroblock in the pixel block of the second image frame that is spatially located at the same position as the reference macroblock set in the first image frame. , Move and reset the search macroblock every even number of pixels, and any two of the reference macroblocks As the weight of the color of the primary color pixels in location it becomes constant, according to the number of pixels of each color of the primary color pixels included in the reference macroblock, by performing the color weighting of the primary colors pixel, the reference macro an image processing apparatus and calculates a motion vector or disparity information by performing block matching for a block and the search macro block.

According to a second technical means, in the imaging apparatus provided with the R, G, B filters in a Bayer arrangement corresponding to each pixel of the imaging element, the first technical processing unit includes the first image processing apparatus, and the Bayer generated by the imaging element. An image pickup apparatus that outputs an image frame having an array of primary color pixel arrangements to the image processing apparatus.

  According to the present invention, for example, based on an image frame output from an imaging device provided with a Bayer array R, G, B filter corresponding to each pixel of the imaging device, a motion vector and disparity information are obtained by a block matching method. Since the search macroblock is set so that the Bayer array primary color pixel arrangement of the reference macroblock is the same as the Bayer array primary color pixel arrangement of the search macroblock, motion vectors and disparity information are detected. Can be detected with high accuracy and at high speed.

1 is a functional block diagram of an image processing apparatus and an imaging apparatus according to a first embodiment of the present invention. It is a figure which shows a reference | standard macroblock. It is a figure which shows a search macroblock. It is a figure which shows a search macroblock. It is a figure which shows a search macroblock. It is a flowchart for demonstrating the setting process of a reference | standard macroblock and a search macroblock, and the detection process of a motion vector. It is a figure which shows a reference | standard macroblock. It is a figure which shows a reference | standard macroblock. It is a figure which shows a reference | standard macroblock. It is a figure which shows a reference | standard macroblock. It is a figure which shows a reference | standard macroblock. It is a figure which shows a reference | standard macroblock. It is a functional block diagram of the image processing apparatus and imaging device which concern on the 2nd Embodiment of this invention. It is a figure which shows a search macroblock. It is a figure which shows the search macroblock after replacing pixel arrangement. It is a figure explaining a prior art.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The configuration in each drawing is exaggerated for easy understanding, and is different from the actual interval and size.
(Embodiment 1)
FIG. 1 is a functional block diagram of an image processing apparatus 1 and an imaging apparatus 2 connected to the image processing apparatus 1 according to the first embodiment of the present invention.
The image processing device 1 includes a motion vector detection unit 11, and the imaging device 2 includes an imaging unit 21 and an image frame memory 22.

First, functional blocks of the imaging device 2 will be described.
The imaging unit 21 includes an imaging element such as a CCD or CMOS, and an R / G / B filter for selecting the wavelength of light is provided in the imaging element. The R, G, and B filters are composed of Bayer array R, G, and B filters, and filters of each color are provided corresponding to each pixel of the image sensor. The imaging unit 21 generates an image frame (image data) by photoelectric conversion in each pixel and outputs the image frame to the image frame memory 22. This image frame has a Bayer array primary color pixel arrangement (Bayer array R, G, B pixel arrangement).
The image frame memory 22 is a memory for storing image frames, and has a storage capacity capable of storing at least two image frames.

Next, functional blocks of the image processing apparatus 1 will be described.
The motion vector detection unit 11 detects a motion vector for a pixel (object) of an image frame stored in the image frame memory 22 of the imaging device 2. As will be described later, the motion vector detection unit 11 can detect not only a motion vector but also parallax information and the like.
Assume that the image frame memory 22 stores a first image frame and a second image frame, which are two consecutive image frames. When the imaging unit 21 captures a moving subject, the subject is captured at a different position in the image.

  When detecting disparity information, two image frames related to the same subject output by two imaging devices arranged at two different viewpoint positions are input to the vector detection unit 11 as a first image frame and a second image frame. Is done.

The reference macroblock setting unit 11 a of the motion vector detection unit 11 sets (defines) a reference macroblock for the first image frame in the image frame memory 22.
That is, the first image having the Bayer array primary color pixel arrangement output from the imaging device 2 provided with the Bayer array R, G, B filter corresponding to each pixel of the image sensor (imaging unit 21). Set the reference macroblock in the frame.

  The search macroblock setting unit 11b sets the second image frame so that the Bayer arrangement primary color pixel arrangement of the reference macroblock set by the reference macroblock setting unit 11a in the first image frame is the same as the Bayer arrangement primary color pixel arrangement. A search macroblock is set for (next image frame). The second image frame also has a primary color pixel arrangement in a Bayer array.

  The difference absolute value sum calculation unit 11c calculates an absolute difference between the pixel value of the pixel of the reference macroblock and the pixel value of the pixel of the search macroblock corresponding to the pixel, and the difference absolute value for all the pixels in the block Is calculated, that is, the difference absolute value sum is calculated and output to the difference absolute value sum evaluation unit 11d. At this time, the coordinate information of the reference macro block and the search macro block is also output to the difference absolute value sum evaluation unit 11d. The coordinate information may be, for example, an xy coordinate at the upper left of each macroblock in the image sensor, an xy coordinate at the center of each macroblock in the image sensor, a relative coordinate with the reference macroblock as the origin, an address on the memory, or the like.

  The difference absolute value sum evaluation unit 11d compares the difference absolute value sums of the macroblocks input from the difference absolute value sum calculation unit 11c, and specifies the position of the search macroblock that minimizes the difference absolute value sum.

Details of the motion vector detection process of the present invention will be described below.
FIG. 2 shows a part of an image frame output by the image sensor of the imaging unit 21.
The letters R, G, and B described in the upper part of the pixel shown in the matrix of FIG. 2 indicate the red pixel, the green pixel, and the blue pixel, respectively, and the four-digit numerical value described in the lower part is pixel position information. The upper two digits indicate the column number, and the lower two digits indicate the row number.
In the Bayer array of FIG. 2, two G pixels are diagonally arranged, and one R pixel and one B pixel are diagonally arranged in a pixel block having a set of two vertical and horizontal pixels (total of four pixels). Array.

First, the reference macroblock setting unit 11a sets a reference macroblock for the first image frame as shown in FIG. In the present embodiment, the size of the reference macroblock is assumed to be 8 pixels vertically and 8 pixels horizontally.
Further, as shown in FIG. 3, the search macroblock setting unit 11b applies the pixel block of the second image frame spatially located at the same position as the reference macroblock set by the reference macroblock setting unit 11a to the first image frame. A search macroblock having the same size as the reference macroblock is set.

  When the setting of the search macroblock is completed, the difference absolute value sum calculation unit 11c sets the pixel value of the pixel of the reference macroblock (hereinafter abbreviated as the pixel value of the reference macroblock) and the pixel of the search macroblock corresponding to the pixel. The absolute value of the difference between the pixel value (hereinafter abbreviated as the pixel value of the search macroblock) is calculated, and the sum of absolute differences S for all the pixels in the block is calculated.

For example, assuming that the pixel value of the reference macroblock is P, the pixel value of the search macroblock is P ′, and the position in the macroblock is the vertical direction i and the horizontal direction j, the difference absolute value sum S is given by (Expression 1) Is calculated by
S = ΣΣ | P _ij −P ′ _ij | (Formula 1)

When obtaining the sum of absolute differences from the reference macroblock of FIG. 2 and the search macroblock of FIG. 3, the pixel value of the upper left G0101 of the reference macroblock of FIG. 2 and the pixel value of the upper left G′0101 of the search macroblock of FIG. The difference in pixel value is calculated for all pixels in the block, such as the difference, the difference between the pixel value of B0201 and the pixel value of B'0201, and the difference of the pixel value of R0102 and the pixel value of R'0102. And the sum is calculated.
That is, in calculating the difference between the pixel value of the reference macroblock in FIG. 2 and the pixel value of the search macroblock in FIG. 3, the difference between the pixel values of the same color is calculated.

  When the calculation of the sum of absolute differences S based on the pixel values of the reference macroblock in FIG. 2 and the pixel values of the search macroblock in FIG. 3 is completed, the difference absolute value sum calculation unit 11c displays the calculation result as the difference absolute value sum evaluation unit 11d. Output to.

  The difference absolute value sum evaluation unit 11d compares the difference absolute value sums of the macroblocks input from the difference absolute value sum calculation unit 11c, and stores the minimum value.

Next, the search macroblock setting unit 11b moves and resets the search macroblock set in the second image frame. At this time, as shown in FIG. 4, the search macroblock setting unit 11b moves rightward from the search macroblock shown in FIG. It is assumed that the pixel is moved by one pixel and reset.
At this time, the Bayer array primary color pixel arrangement of the reference macroblock shown in FIG. 2 is different from the Bayer array primary color pixel arrangement of the search macroblock shown in FIG. For example, the upper left G0101, the right B0201, and the lower R0102 of the reference macroblock in FIG. 2 correspond to the upper left B′0201, G′0301, and G′0202 in FIG. The primary color pixel arrangement is different.

Since the difference absolute value sum S of both macroblocks calculated based on the pixel values of different colors compares different colors even when the difference absolute value sum is minimized, there is a possibility that it is a false detection. Get higher.
Therefore, the search macroblock setting unit 11b according to the present invention sets the second search macroblock so that the Bayer array primary color pixel layout of the reference macroblock is the same as the Bayer array primary color pixel layout of the search macroblock. Set to image frame.
In the above example, the search macroblock setting unit 11b sets the search macroblock of FIG. 5 shifted by two pixels (even pixels) in the horizontal direction from the search macroblock shown in FIG. 3 in the second image frame.

At this time, the reference macroblock upper left G0101, its right B0201, and its lower R0102 in FIG. 2 correspond to the search macroblock upper left G′0301, B′0401, and R′0302 in FIG. The primary color pixel arrangement in the Bayer array of blocks is the same.
When the setting of the search macroblock is completed, the difference absolute value sum calculation unit 11c calculates the difference absolute value sum S based on the pixel value of the reference macroblock in FIG. 2 and the pixel value of the search macroblock in FIG. .

  Thereafter, the search macroblock setting unit 11b performs a search so that the Bayer array primary color pixel arrangement of the reference macroblock set in the first image frame is the same as the Bayer array primary color pixel arrangement of the search macroblock. Set the macroblock to the second image frame. In the above example, the search macroblock is moved and reset every even pixel.

  An example of the reference macroblock and search macroblock setting processing and motion vector detection processing described above will be described with reference to the flowchart of FIG. The motion vector detection process described in the flowchart of FIG. 6 is based on the assumption that the motion vector detection process according to the present invention is executed using software, for example, and the motion according to the present invention is performed using hardware. When performing vector detection processing, the motion vector detection processing may be executed in parallel.

  The reference macroblock setting unit 11a sets a reference macroblock in the first image frame stored in the image frame memory 22 of the imaging device 2 (step S1).

  Next, the search macroblock setting unit 11b has a second position that is spatially the same as the reference macroblock set in the first image frame with respect to the next image frame (second image frame) of the first image frame. A search macroblock having the same size as the reference macroblock is set in the pixel block of the image frame (step S2). Since the search macro block and the reference macro block are spatially at the same position, the color arrangement of each block is the same.

  The difference absolute value sum calculation unit 11c calculates a difference absolute value sum S between the pixel value of the reference macroblock and the pixel value of the search macroblock, and the difference absolute value sum evaluation unit 11d along with the coordinate information of each macroblock. (Step S3).

  Next, the search macroblock setting unit 11b sets the search macroblock that has been set so that, for example, the Bayer array primary color pixel arrangement of the reference macroblock is the same as the Bayer array primary color pixel arrangement of the search macroblock. To the right by every even pixel (even pixel from top to bottom) and reset (step S4).

  Then, after setting the search macroblock, as described in step 3, the sum of absolute differences S between the pixel value of the reference macroblock and the pixel value of the search macroblock is calculated, and the calculation result is used as the coordinate information of each macroblock. At the same time, it is output to the difference absolute value sum evaluation unit 11d.

  The search macroblock setting process and the difference absolute value sum calculation process described in step S4 are repeatedly executed for the predetermined area of the second image frame (steps S3 to S5), and the predetermined area of the second image frame is determined. When the setting process and the calculation process are completed (step S5 / YES), the process of step S6 is executed. The predetermined area is determined based on an algorithm, accuracy, processing amount, and the like. For example, when searching for an object having a maximum moving distance of 32 pixels between frames, a range of ± 32 pixels is predetermined with respect to the target pixel. A region may be used. If the setting process is not completed (step S5 / NO), the process returns to step S3.

  In step S6, the difference absolute value sum evaluation unit 11d specifies a search macroblock corresponding to the minimum difference absolute value sum. Then, the motion vector detection unit 11 regards the specified search macroblock as a destination macroblock of the reference macroblock, and sets the coordinate information of the reference macroblock and the specified minimum difference absolute value search macro Based on the block coordinate information, the motion vector of the reference macroblock is detected.

  When the reference macroblock setting process is completed for the predetermined area of the first image frame (step S7 / YES), the process ends. In the next step, a reference macro block is set for the second image frame, and a search macro block is set for the third image frame.

  If the reference macroblock setting process is not completed for the predetermined area of the first image frame (step S7 / NO), the process returns to step S1. In this step, the reference macroblock set in the first image frame is moved by, for example, one pixel from left to right (one pixel from top to bottom) and reset. Note that the movement amount (moving pixel) of the reference macroblock is appropriately determined depending on the angle of view of the captured image, the resolution of the image sensor, the required motion vector resolution, and the like.

  As described in step S4, when a motion vector is detected with the color of the pixel of the reference macroblock and the color of the pixel of the search macroblock corresponding to the reference macroblock being the same color, the same as when using luminance information. It is possible to improve the detection accuracy while maintaining the processing amount.

  Further, since the search macroblock is not moved to a position where there is a high possibility of erroneous detection but is moved so that the pixel colors are the same, there is also an effect that the calculation amount at the time of detecting the motion vector becomes half or less.

Next, the size of the macroblock will be described. Although the size of the reference macroblock described in FIG. 2 is 8 pixels in the vertical direction and 8 pixels in the horizontal direction, reference macroblocks of various sizes can be set.
For example, as shown in FIGS. 7 and 8, the size of the reference macroblock can be set to 8 pixels vertically and 7 pixels horizontally. However, in this case, the following problem occurs, and thus correction processing is executed in calculating the sum of absolute differences.

  The size of the reference macroblock shown in FIG. 7 is 8 pixels vertically and 7 pixels horizontally. At this time, the reference macroblock shown in FIG. 8 is compared with the reference macroblock shown in FIG. The reference macroblock shown in FIG. 8 is a reference macroblock obtained by shifting the reference macroblock shown in FIG. 7 horizontally by one pixel.

The number of green pixels in the reference macroblock in FIG. 7 is 28, the number of red pixels is 16, and the number of blue pixels is 12. Similarly, the number of green pixels in the reference macroblock of FIG. 8 is 28, the number of red pixels is 12, and the number of blue pixels is 16.
Therefore, as described above, if the sum of absolute differences is calculated using the pixel value of each pixel in the reference macroblock and the pixel value of each pixel in the search macroblock corresponding to the reference macroblock, the reference in FIG. Since the macroblock has the same number of green pixels, more red pixels, and fewer blue pixels than the reference macroblock in FIG. 8, the red pixels are weighted.
That is, even for adjacent reference macroblocks, the sum of absolute differences with different weights is calculated, and the detection result of the motion vector of the reference macroblock may be different.

  When a motion vector is detected based on such a reference macroblock and a search macroblock corresponding to the reference macroblock, the motion vector seems to be detected correctly. When the pixel values of the blocks take similar values, the motion vector cannot be detected correctly, and a state similar to false detection with a variation in values is seen as a whole.

This is the same even if the number of vertical pixels in the reference macroblock is odd and the number of horizontal pixels is even. For example, the reference macro shown in FIGS. 9 and 10 has 9 vertical pixels and 8 horizontal pixels. Compare blocks.
The number of green pixels in the reference macroblock in FIG. 9 is 36, the number of red pixels is 16, and the number of blue pixels is 20. The number of green pixels in the reference macroblock in FIG. 10 is 36, the number of red pixels is 16, and the number of blue pixels is 20. Therefore, the red and blue weights for green are different.

This is the same even if the number of vertical pixels in the reference macroblock is odd and the number of horizontal pixels is odd. For example, the reference macroblock shown in FIGS. 11 and 12 has 7 vertical pixels and 7 horizontal pixels. Compare
The number of green pixels in the reference macroblock of FIG. 11 is 25, the number of red pixels is 12, and the number of blue pixels is 12. The number of green pixels in the reference macroblock in FIG. 12 is 24, the number of red pixels is 9, and the number of blue pixels is 16. Therefore, the weights for all colors are different.

  From the above, it can be seen that the size of the reference macroblock whose weighting does not change for all colors is the case where the number of vertical and horizontal pixels is an even number. Therefore, by setting the reference macroblock so that the vertical and horizontal pixels are even pixels, the color weighting does not change depending on how the reference macroblock is set, and a good calculation result can be obtained.

However, when setting the reference macroblock, if you want the vertical and horizontal widths of the reference macroblock to be symmetrical with respect to the target pixel of the image frame (the center pixel of the reference macroblock is aligned with the target pixel), etc. There is also a case where it is desired to configure.
Therefore, if the number of vertical and horizontal pixels in the macroblock is odd, the difference absolute value sum is calculated by executing the correction calculation process on the difference absolute value calculation result in consideration of the number of color pixels. Is calculated.
For example, the reference macroblock shown in FIGS. 11 and 12 having 7 pixels in the vertical direction and 7 pixels in the horizontal direction will be described. In the case of the reference macroblock shown in FIG. 11, the difference absolute value sum calculation unit 11c calculates the difference between the green pixels. The absolute value calculation result is divided by the number of green pixels in the block (25), the difference absolute value calculation result of the red pixel is divided by the number of red pixels in the block (12), and the difference absolute value calculation result of the blue pixel is calculated as the number of blue pixels in the block. Divide by (12) to calculate the sum of each division value.

  In the case of the reference macroblock shown in FIG. 12, the difference absolute value sum calculation unit 11c divides the difference absolute value calculation result of the green pixel by 24, divides the difference absolute value calculation result of the red pixel by 9, The difference absolute value calculation result is divided by 16 to calculate the sum of the divided values. By doing so, the weighting of each color can be made uniform, and the color weighting does not change depending on the size of the reference macroblock, and a good calculation result can be obtained.

  If search macroblocks are set so that the number of vertical and horizontal pixels of a macroblock is an even number, color weighting does not change and good calculation is possible.

(Embodiment 2)
Next, a second embodiment of the present invention will be described with reference to the drawings. Note that portions having the same functions as those of the first embodiment are denoted by the same reference numerals, and description thereof is omitted.
FIG. 13 is a functional block diagram of the image processing device 1 ′ and the imaging device 2 connected to the image processing device 1 ′ according to the second embodiment of the present invention.

  The search macroblock setting unit 11b of the image processing apparatus 1 according to the first embodiment is configured so that the Bayer array primary color pixel layout of the reference macroblock is the same as the Bayer array primary color pixel layout of the search macroblock. The search macroblock was set to the second image frame.

  However, even if the Bayer array primary color pixel arrangement of the reference macroblock is different from the Bayer array primary color pixel arrangement of the search macroblock, the reference macroblock Bayer of the reference macroblock can be changed by switching the arrangement of the set search macroblock pixels. The primary color pixel arrangement in the array and the Bayer array primary color pixel arrangement of the search macroblock can be made the same.

The search macroblock setting unit 11b ′ has the same size as the reference macroblock in the pixel block of the second image frame spatially located at the same position as the reference macroblock set by the reference macroblock setting unit 11a in the first image frame. A search macroblock is set, and the search macroblock is moved and reset every pixel or odd pixel.
At this time, unlike the first embodiment, the Bayer array primary color pixel arrangement of the reference macroblock set in the first image frame may be different from the Bayer array primary color pixel arrangement of the search macroblock.

  For example, the search macroblock setting unit 11b ′ of the image processing device 1 ′ according to the second embodiment of the present invention sets the search macroblock corresponding to the reference macroblock shown in FIG. 2 as the second image as shown in FIG. Suppose you set it to a frame. In this case, the Bayer array primary color pixel arrangement of the reference macroblock is different from the Bayer array primary color pixel arrangement of the search macroblock.

  Therefore, the pixel arrangement replacement unit 11e of the image processing apparatus 1 ′ of the present invention differs between the set (re-set) Bayer arrangement primary color pixel arrangement of the search macroblock and the Bayer arrangement primary color pixel arrangement of the reference macroblock. If this is detected, the primary color pixel arrangement of the search macroblock is made the same as the primary color pixel arrangement of the search macroblock Bayer array.

  Note that, by detecting that the search macroblock has moved by an odd number of pixels from the reference macroblock, it can be considered that the primary color pixel arrangement of both macroblocks is different.

  In the example of FIG. 14, the pixel arrangement replacing unit 11e replaces the red pixel and the blue pixel of the search macro block.

  FIG. 15 shows a search macroblock when the red and blue pixels of the search macroblock in FIG. 14 are replaced. In the search macroblock shown in FIG. 15, R′0302 and B′0203 are interchanged so as to correspond to the pixel arrangement of red pixels and the pixel arrangement of blue pixels of the reference macroblock shown in FIG. Similarly, R′0502 and B′0403 and R′0702 and B′0603 are interchanged.

  Thereafter, as described above, the motion vector detection unit 11 ′ detects a motion vector by performing block matching on the reference macroblock and the search macroblock.

By doing in this way, the color of the pixel of a reference | standard macroblock and a search macroblock becomes mutually the same, and a difference absolute value sum is correctly calculated.
In this way, in the search macro block in which the odd-numbered pixels are shifted vertically and the odd-numbered pixels are shifted from the reference macro-block, the primary color pixel arrangement is changed so that the colors of the pixels of each macro-block are the same, so that the correct difference is obtained. The absolute value sum can be calculated.

  In the image processing apparatus according to the second embodiment, the positions of the red pixel and the blue pixel are switched without changing the position of the green pixel. As described above, if the number of red pixels is different from the number of blue pixels, an error occurs in the sum of absolute differences. Therefore, the reference macroblock has the same number of red pixels and blue pixels in the macroblock. It is preferable to determine the size of the search macroblock.

  According to the image processing apparatus of the second embodiment, the difference absolute value sum can be correctly calculated even when the Bayer array primary color pixel layout of the reference macroblock is different from the Bayer array primary color pixel layout of the search macroblock.

  Further, according to the image processing apparatus of the second embodiment, the resolution of motion vectors and parallax can be improved. In the first embodiment, the search macroblock is moved every even number of pixels. In this case, the minimum movement amount of the search macroblock is a distance of two pixels in the vertical and horizontal directions. On the other hand, in this embodiment, the minimum movement amount of the search macroblock is a distance of one pixel in the vertical and horizontal directions. Therefore, the distance between search macroblocks can be set to √2. Therefore, the movement of the search macroblock becomes finer, and the calculation resolution of motion vectors, parallax, etc. can be improved.

In this embodiment, the motion vector is detected using the sum of absolute differences (ΣΣ | P _ij −P ′ _ij |), but the motion is calculated using the sum of squares of differences (ΣΣ (P _ij −P ′ _ij ) ² ). Vectors may be detected.

In the present embodiment, motion vector detection has been described, but the present invention can be applied to any processing that uses block matching.
For example, the present invention may be applied when calculating the distance from the subject to the imaging device based on the parallax information. In this case, for example, if block matching according to the present invention is performed on two image frames related to the same subject output by two imaging devices arranged at two different viewpoint positions, the parallax with respect to the same subject can be detected. The distance from the subject to the imaging device can be obtained.
In addition, various methods proposed conventionally can be used for detection of parallax.

  In addition, an ordinary image pickup apparatus is provided with an optical system for guiding light to the image pickup device. When uneven illuminance occurs in the image pickup device, processing such as shading correction is performed on the image data. After being performed, detection accuracy can be improved by performing block matching. Furthermore, detection accuracy can be improved by performing block matching after performing correction processing such as unevenness between pixels of the image sensor and a flaw defect.

The image sensor 2 and the image processing apparatus 1 may be integrated. In this case, the imaging device 2 includes the image processing device 1, and outputs an image frame having a Bayer array-like primary color pixel arrangement generated by the imaging unit 21 of the imaging device 2 to the image processing device 1. Note that the image sensor 2 and the image processing apparatus 1 ′ may be integrated.
Further, the frame memory 22 of the imaging device 22 may be provided in the image processing devices 1 and 1 ′.

DESCRIPTION OF SYMBOLS 1,1 '... Image processing apparatus 11, 11' ... Motion vector detection part, 11a ... Reference | standard macroblock setting part, 11b ... Search macroblock setting part, 11c ... Difference absolute value sum calculation part, 11d ... Difference absolute value sum Evaluation part, 11e ... Pixel arrangement replacement part, 2 ... Imaging device, 21 ... Imaging part, 22 ... Image frame memory.

Claims (2)

A reference macroblock setting unit for setting a reference macroblock in a first image frame having a Bayer array-like primary color pixel arrangement;
To be the same Bayer arrangement shaped primary color pixels arranged as Bayer array shaped primary color pixel arrangement of the reference macro-blocks set in the first image frame, the search macro block setting for setting a search macroblock with respect to the second image frame With
The search macroblock setting unit has the same size as the reference macroblock in the pixel block of the second image frame spatially located at the same position as the reference macroblock set by the reference macroblock setting unit in the first image frame. The search macroblock is set, and the search macroblock is moved and reset every even pixel,
According to the number of pixels of each color of the primary color pixels included in the reference macroblock, each color of the primary color pixel is set so that the color weight of the primary color pixel is constant at any two positions of the reference macroblock. by performing the weighting, an image processing apparatus characterized by calculating a motion vector or disparity information by performing block matching on the reference macro block and the search macro block. In an imaging apparatus provided with a Bayer array R, G, B filter corresponding to each pixel of the imaging element,
An image pickup apparatus comprising the image processing apparatus according to claim 1 , wherein an image frame having a Bayer array primary color pixel arrangement generated by the image pickup device is output to the image processing apparatus.

Images