JP2009129318A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a computational quantity for performing sub-pixel calculation in stereo processing. <P>SOLUTION: A stereo matching part 2 uses each partial area in one image as a correlation source to search a correlation destination of individual partial area in the other image and calculates an amount of deviation of the correlation destination against the coordination source as a pixel value expressed by integral multiple of a pixel. An object area specification part 4 dynamically specifies an object area where an object exists on an image plane. A sub-pixel processing part 3 calculates a sub-pixel value higher in resolution than the pixel value only in the partial area corresponding to an object area specified by the object area specification part 4 in terms of a position. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that performs stereo matching for each region using a pair of images, and more particularly to sub-pixel calculation of a correlation destination shift amount with respect to a correlation source.

  2. Description of the Related Art Conventionally, there is known a method for calculating a shift amount of a region having a correlation with each other at a sub-pixel level in stereo processing in which stereo matching using a pair of images is performed for each region. For example, Patent Document 1 discloses a method for calculating a subpixel component by linear approximation of a temporary corresponding point that gives pixel-level parallax, an adjacent point immediately before, and an adjacent point immediately after that. . In addition to Patent Document 1, methods such as Patent Documents 2 to 5 are also known. As is well known, if the sub-pixel calculation is performed by stereo processing, the position of the object in the real space can be specified with higher resolution. Therefore, it is possible to improve the ranging accuracy of a preview sensor or the like using a stereo camera.

JP 2000-283755 A JP 2004-219142 A JP 2003-150939 A JP 2003-97944 A JP-A-5-265547

  However, in Patent Documents 1 to 5 described above, since the sub-pixel calculation in the stereo processing is performed over the entire image, there is a disadvantage that the calculation amount increases.

  Therefore, an object of the present invention is to reduce the amount of calculation in subpixel calculation in stereo processing.

  In order to solve such a problem, the present invention has a stereo matching unit, an object region specifying unit, and a subpixel processing unit, and a partial region in which stereo matching using a pair of images is set on an image plane Provided is an image processing apparatus which is performed every time. The stereo matching unit uses each partial region in one image as a correlation source, and searches for a correlation destination of each partial region in the other image. Then, the stereo matching unit calculates the deviation amount of the correlation destination with respect to the correlation source as a pixel value represented by an integer multiple of the pixel. The object region specifying unit dynamically specifies an object region where an object exists on the image plane. The sub-pixel processing unit calculates a sub-pixel value having a resolution higher than the pixel value by limiting to a partial region that corresponds in position to the object region specified by the object region specifying unit.

  Here, in the present invention, the object region specifying unit is close enough to be regarded as the same object on the two-dimensional plane defined by the set of pixel values calculated by the stereo matching unit, and is positioned positionally. It is preferable to identify the object region by grouping adjacent pixel values. Alternatively, an area that matches the luminance pattern as an object on the luminance plane of the image may be specified as the object area.

  In the present invention, it is preferable that the subpixel processing unit sets a region for calculating a subpixel value in the current frame based on the object region in the previous frame specified by the object region specifying unit. Further, when a plurality of object areas are detected, the subpixel processing unit sets a priority order for each of the recording object areas based on a preset priority order rule, and the subpixels are set according to the set priority order. A value may be calculated.

  According to the present invention, an object region where an object exists on an image plane is dynamically specified, and sub-pixel calculation is performed only in this region. Thereby, compared with the case where the subpixel calculation is performed over the entire image, the amount of calculation can be reduced by a limited amount of the area in which the subpixel calculation is performed.

(First embodiment)
FIG. 1 is a block diagram of the image processing apparatus according to the first embodiment. The image processing apparatus 1 is mainly configured by a stereo matching unit 2, a subpixel processing unit 3, and an object region specifying unit 4. The image processing apparatus 1 is used as, for example, a part of a preview sensor that monitors the front of a vehicle with a stereo camera, specifically, a preprocessing unit of a recognition unit that recognizes an object. In this case, as a pair of images input to the stereo matching unit 2, a stereo image output from a stereo camera (not shown), that is, a pair of images that are simultaneously imaged and output by a pair of cameras arranged with a predetermined camera baseline length. A pair of images is used. Further, the image processing apparatus 1 can be applied to an obstacle detection apparatus using a stereo camera, a terrain recognition apparatus, a railroad crossing monitoring apparatus, an altitude measuring apparatus for a flying object, and the like in addition to the in-vehicle preview sensor. Furthermore, the image processing apparatus 1 is not limited to a stereo camera, and can be used as a pre-processing unit of a detection unit that detects, for example, an amount of movement or speed of an object as an optical flow. In this case, an object is repeatedly imaged at a predetermined interval with a monocular camera, and an image captured at a certain timing and an image captured at a different timing are input to the stereo matching unit 2 as a pair of images. .

  In the following description, one of the pair of images is referred to as a “reference image”, and the other is referred to as a “comparison image”. In the case of a stereo camera, as an example, among the images taken at the same time, an image of the main camera is used as a reference image, and an image of the sub camera is used as a comparison image. In the case of a monocular camera, among images captured at different timings in time series, for example, the earlier imaging timing is used as a reference image, and the later imaging timing is used as a comparison image. These images are digital images having a predetermined luminance gradation (for example, gray scale of 256 gradations), and one frame image is defined as a two-dimensional planar set of luminance data. An image plane that defines one frame is expressed in an ij coordinate system, with the lower left corner of the image as the origin, the horizontal direction as the i axis, and the vertical direction as the j axis.

  As shown in FIG. 5A, the stereo matching unit 2 calculates the pixel value D by performing stereo matching using the reference image and the comparison image as inputs. Here, as shown in FIG. 5B, the pixel value D is calculated for each partial region in the image plane defined by the reference image, and is associated with the position (i, j) on the image plane. . One pixel value D is calculated for each pixel block (for example, 4 × 4 pixels) as a partial area constituting a part of the reference image, and the pixel block is a calculation unit.

  FIG. 2 is an explanatory diagram of pixel blocks set in the reference image. For example, when the reference image is composed of 200 × 512 pixels, 50 × 128 pixel blocks PBij are set as partial areas in one frame image. In the case of a stereo camera, “parallax” as a pixel value D is calculated for each pixel block PBij that is a partial region. As is well known, the parallax is a horizontal shift amount of the pixel block PBij that is a calculation unit thereof, and has a large correlation with the distance to the object projected in the pixel block PBij. The closer the object projected in the pixel block PBij is to the stereo camera, the larger the parallax D of the pixel block PBij, and the smaller the object, the smaller the parallax D (if the object is infinitely far away, the parallax D Becomes 0).

  When the pixel value D of a certain pixel block PBij (correlation source) is calculated, a region (correlation destination) having a correlation with the luminance characteristic of the pixel block PBij is specified in the comparison image. In the case of a stereo camera, the distance from the stereo camera to the object is reflected as a horizontal shift amount between the reference image and the comparison image. Therefore, when searching for the correlation destination in the comparison image, it is not necessary to search the entire comparison image, and it is only necessary to search on the same horizontal line (epipolar line) as the j coordinate of the pixel block Pij as the correlation source. The stereo matching unit 2 sequentially shifts the correlation between the correlation source and the correlation destination candidates while shifting one pixel at a time on the epipolar line within a predetermined search range set based on the i coordinate of the correlation source. Evaluate (stereo matching).

Various methods for determining the correlation between two pixel blocks are known, but here, the city block distance H is used. Formula 1 shows a basic form of the city block distance H. p1ij is the luminance value of the ij-th pixel in one pixel block, and p2ij is the ij-th luminance value in the other pixel block. The city block distance H is the sum of the differences (absolute values) of the luminance values p1ij and p2ij corresponding to each other in the entire pixel block, and the smaller the difference, the greater the correlation between the two pixel blocks.
(Formula 1)
H = Σ | p1ij−p2ij |

  Basically, of the city block distances H calculated for each pixel block existing on the epipolar line, in principle, the pixel block having the smallest value is determined as the correlation destination. The amount of deviation between the correlation destination and the correlation source specified in this way is the pixel value D. Due to the characteristics of stereo matching, the pixel value D representing the amount of deviation of the correlation destination relative to the correlation source is expressed as an integer multiple of the pixel.

  By the above method, the stereo matching unit 2 sequentially calculates the pixel value D for each reference pixel block constituting one frame of the reference image, and outputs it to the sub-pixel processing unit 3. At that time, as shown in FIG. 3, the city block distance H1 (= Hmin) relating to the point p1 giving the pixel value D, the city block distance H0 of the adjacent point p0, and the adjacent point p2 immediately thereafter The city block distance H2 is also output to the sub-pixel processing unit 3.

The basic form of the city block distance is as described above, but there are many variations in the specific calculation formula. For example, instead of normal stereo matching, average value difference matching may be performed. The average value difference matching has a merit that low-frequency noise can be effectively removed because only a high-frequency component of a captured image is a matching target and has an action equivalent to a high-pass filter. In the average value difference matching, the city block distance H is calculated according to Equation 2. Here, Aave is the average luminance value of the reference pixel block A, and Bave is the average luminance value of the comparison pixel block B. That is, in the average value difference matching, a difference between a value obtained by subtracting the luminance average value Aave from the luminance value aij of the reference pixel block A and a value obtained by subtracting the luminance average value Bave from the luminance value bij of the comparison pixel block B ( (Absolute value) is defined as the sum of the entire pixel block. The details of the average value difference matching processing are disclosed in, for example, Japanese Patent Application Laid-Open No. 11-234701, and should be referred to if necessary.
(Formula 2)
H = Σ | (aij−Aave) − (bij−Bave) |
Aave = Σaij / (I × J)
Bave = Σbij / (I × J)

Further, in the case where importance is attached to the luminance values at the center of the pixel blocks A and B (for example, the same applies to a22, a23, a32, a33, and bij), the weight coefficient is added to the absolute value of the luminance difference as shown in Equation 3. The city block distance H may be calculated by multiplying by wij (weighting matching). In this case, the weighting coefficient wij is set to a larger value as it goes inside the pixel blocks A and B. Details of the weighting matching processing are disclosed in, for example, Japanese Patent Application No. 2001-063290.
(Formula 3)
H = Σwij | aij−bij |

  The sub-pixel processing unit 3 performs interpolation by sub-pixel processing on the pixel value D having the resolution of one pixel unit generated in the stereo matching unit 2, and calculates the sub-pixel value Ds having the resolution of one pixel unit or less. To do. In the case of a stereo camera, it is known from the principle of triangulation that an essential characteristic is that the distance measurement resolution inevitably decreases as the distance to the object projected in the captured image increases. Therefore, it is necessary to acquire information with a resolution higher than that of the pixel value D, and as one means, a sub-pixel value Ds having a decimal pixel of one pixel or less is used.

  Various methods have been proposed as the calculation method of the subpixel value Ds, and any of the calculation methods may be used. Here, two calculation methods are exemplified.

<Calculation Method 1> Linear Approximation on IH Plane In this method disclosed in Japanese Patent Laid-Open No. 2000-283753, as shown in FIG. 3, I (deviation amount) -H (city block distance) plane In the above discrete distribution, the sub-pixel value Ds is calculated by linear approximation using three points: the provisional corresponding point p1 that gives the pixel value D, the adjacent point p0 immediately before it, and the adjacent point p2 immediately after that. Assuming that the size of one pixel is infinitely small, the distribution of the city block distance H in the image plane (i, j) is continuous as shown in FIG. When this is regarded as a one-dimensional distribution of the deviation detection direction, a continuous distribution as shown by a broken line in FIG. 3 is obtained. As this continuous distribution shows, it becomes symmetrical around the true corresponding point psub (the point that takes the original minimum value in the continuous distribution of the city block distance H) giving the subpixel value Ds. However, it is known that the symmetry is logically guaranteed in a very narrow range and within ± 1 pixel of the corresponding point psub, that is, within the range of the points p0 to p2. In view of the symmetry of the city block distance H, when the immediately adjacent city block distance H0 is larger than the immediately adjacent city block distance H2 (case in FIG. 3), the corresponding point psub is the temporary corresponding point p1. It can be seen that it is located on the right side (the point side where the city block distance is small) from the point (the point that becomes the minimum city block distance in the distribution of discrete city block distances).

  In this method, two straight lines L1 and L2 are used to identify one point that can be regarded as the corresponding point psub within the right range is to (is + 1) of the temporary corresponding point p0. Specifically, first, of the adjacent points p0 and p2, the adjacent point p0 that gives the larger city block distance H is selected, and a straight line L1 (inclination) connecting the temporary corresponding point p1 and the selected adjacent point p0. = M) is set. Next, a straight line L2 is set which is symmetrical with the straight line L1 with respect to the vertical axis (slope = −m) and passes through the adjacent point p2 which is not selected. Then, the intersection of two straight lines L1 and L2 that are line symmetrical with each other is calculated. The I coordinate value of this intersection is the sub-pixel level deviation amount with respect to the pixel value D, that is, the sub-pixel component S. The subpixel value Ds is obtained by adding the subpixel component S to the pixel value D.

<Calculation Method 2> Stereo Matching with Interpolated Luminance The present method disclosed in Japanese Patent Laid-Open No. 2003-150939 is similar to Calculation Method 1 in that it focuses on the symmetry of the city block distance H described above. The calculation method is different, and stereo matching based on interpolation luminance is used. Specifically, when it is determined that the corresponding point psub is within the right range is to (is + 1) of the temporary corresponding point p0, luminance interpolation data is generated within this range. This interpolation data is generated by, for example, linearly interpolating the luminance values of actual pixels in units of 4 × 4 pixel blocks. When linear interpolation is performed within the range of pixel shift amounts is to i (s + 1), that is, between the comparison pixel block B [Is] and the comparison pixel block B [Is + 1], the interpolation of Equation 4 is performed. An equation is used. Here, B (is) ij and B (is + 1) ij are the luminance values of the pixel groups constituting the comparison pixel blocks B [is] and B [is + 1], respectively. L is an interpolation resolution (constant), I is a parameter (0 ≦ I ≦ L) indicating the position of the interpolation pixel, and takes an integer value.
(Formula 4)
bij = B (is) ij + I * (B (is + 1) ij-B (is) ij) / L

  In this way, the position of the imaginary pixel (interpolation pixel) to be obtained by interpolation is moved in accordance with the interpolation resolution (that is, in units of 1 / L pixel), and the number of virtual pixels corresponding to the interpolation resolution L is virtual. A comparative pixel block is generated. For example, when the interpolation resolution L is 256, 255 comparison pixel blocks are obtained as the interpolation data. Note that the comparison pixel blocks B [is-1], B [is], and B [is + 1] are shifted from each other by one pixel in the horizontal line direction, and thus in the comparison pixel block B [is-1]. The pixel in i row and j column, the pixel in i row and j-1 column in comparison pixel block B [is], and the pixel in i row and j-2 column in comparison pixel block B [i + 1] are the same. (For example, B (is-1) 13 = B (is) 12 = B (is + 1) 11).

  Sub-pixel level stereo matching is performed using this interpolation data. That is, for each comparison pixel block constituting the interpolation data, stereo matching with the reference pixel block is performed, and the shift amount is calculated. This stereo matching itself is the same as the pixel level stereo matching. The calculated shift amount is a shift based on the position of the pixel value D (= is), and is in 1 / L pixel unit (that is, decimal pixel unit). The subpixel component S is calculated by dividing the shift amount calculated in this way by the interpolation resolution L. The subpixel component S corresponds to an offset value between the pixel value D (= is) and the subpixel value Ds (= isub). Therefore, if the subpixel component S is added to the pixel value D, a subpixel value Ds having a resolution of one pixel or less is calculated.

  The sub-pixel value Ds calculated in this way is output to a processing unit (not shown) in the subsequent stage. The processing unit recognizes an object to be monitored (for example, a preceding vehicle) or detects an optical flow of the object by using the sub-pixel value Ds as necessary.

  Based on the pixel value D calculated by the stereo matching unit 2, the object region specifying unit 4 dynamically specifies the object region where the object exists on the image plane. Specifically, the object is specified by grouping pixel blocks adjacent to each other on the image plane. First, pixel values D that are positioned adjacently on the top, bottom, left, and right of the image plane are sequentially grouped when the amount of change is within a predetermined threshold, in other words, the values that are close enough to be regarded as the same object. It will become. Thereby, a large group is formed for a large object (A, B, C in FIG. 5C). Next, the size (area) of the grouped pixel block group is calculated. Then, it is determined whether or not the group size is larger than a predetermined threshold value. A group whose size is larger than the threshold value (groups A and C in FIG. 5C) is regarded as highly reliable as a region where an object is projected there, and is extracted as an object region. On the other hand, the group whose size is equal to or smaller than the threshold value (B in FIG. 5C) is considered to have low reliability and is not extracted as an object region. Information regarding the object areas A and C thus identified is notified to the sub-pixel processing unit 3. The details of the grouping of pixel blocks as described above are disclosed as group filters in JP-A-6-266828 and JP-A-10-285582, so refer to them if necessary.

  Note that the object area specifying unit 4 may specify the object area based on the luminance of one image instead of the process based on the pixel value calculated from the pair of images described above. In this case, on the luminance plane of the image of one frame, an area that matches the luminance pattern as the object is extracted as the object area.

  As shown in FIG. 5D, the subpixel processing unit 3 limits the subpixel values to the object areas A and C specified by the object area specifying unit 4 instead of the entire image of one frame. Ds is calculated. The pixel blocks for which the subpixel value Ds is calculated are pixel blocks included in the object areas A and B specified by the object area specifying unit 4.

  As described above, according to the present embodiment, an object region where an object exists on the image plane is dynamically specified, and sub-pixel calculation is performed only in this region. Thereby, compared with the case where the subpixel calculation is performed over the entire image, the amount of calculation can be reduced by a limited amount of the area in which the subpixel calculation is performed.

(Second Embodiment)
The identification of the object area and the subpixel calculation based on the area limitation based on the object area may be a process completed within one frame, or may be a process over a plurality of consecutive frames. In the case of using the former form, that is, the object region specified in a certain frame in the subpixel calculation in the same frame, the identification of the object region and the subsequent subpixel calculation are performed within the processing period of one frame. It must be completed, and time constraints are severe. Therefore, if this embodiment related to the latter mode, that is, an object region specified in a certain frame is used in subpixel calculation in a later frame, such a time restriction can be relaxed.

  FIG. 6 is a timing chart of frame processing according to the present embodiment. This figure shows processing for two frames. The nth frame (hereinafter referred to as “n frame”) in the period t1 to t3, and the (n + 1) th frame (hereinafter referred to as “(n + 1) frame”) in the period t3 to t5. Are each processed. First, at the start timing t1 of n frames, stereo matching for n frames is started. Then, at the timing t2 when the stereo matching is finished, the identification of the object region for the n frame is started, and in parallel with this process, the subpixel calculation for the n frame is also started. However, since the object region related to the n frame is not specified at the start of the subpixel calculation, the object region specified for the immediately preceding (n−1) th frame (hereinafter referred to as “(n−1) frame”). Applies. In other words, the n-pixel sub-pixel calculation is performed only on the object region specified by the (n−1) frame. As a result, it is possible to carry out both the identification of the object region relating to the n frames and the subpixel calculation in parallel.

  Next, at the start timing t3 of the (n + 1) frame, stereo matching for the (n + 1) frame is started. Then, at the timing t4 when the stereo matching is completed, the identification of the object region for the (n + 1) frame is started, and in parallel with this process, the subpixel calculation for the (n + 1) frame is also started. Is done. The subpixel calculation of the (n + 1) frame is performed only on the object region specified in the immediately preceding n frame. As a result, it is possible to perform both the object region specification and the sub-pixel calculation for (n + 1) frames in parallel.

  According to the present embodiment, in addition to the same effects as those of the first embodiment described above, the object region specification and the sub-pixel calculation are performed in parallel, so that the processing time is compared with the first embodiment. Can be shortened.

(Third embodiment)
As in the second embodiment, when the subpixel calculation for the current frame is performed using the object region specified in the previous frame, the previous object region may be applied as it is. The position of the object region in the current frame may be estimated according to the change over time. FIG. 7 is an explanatory diagram of object region estimation according to the present embodiment. FIG. 4A shows an image of (n−2) frames, and FIG. 4B shows an image of (n−1) frames. In this case, as shown in FIG. 5C, the (n-2) frame and the (n-1) frame are determined based on the position (movement amount) and size (stretching rate) of the object region. A movement vector M indicating a positional change is calculated. Based on the movement vector M, the position of the object region in the current cycle n is specified.

  According to this embodiment, in addition to the same effects as those of the first and second embodiments, there is an effect that the object region can be accurately identified even when the object moves greatly.

(Fourth embodiment)
For example, as shown in FIG. 8, when there are two preceding vehicles ahead of the host vehicle, two object regions are specified in one frame image. In such a case, it is preferable to set a priority order for each object region and perform sub-pixel calculation accordingly in order to achieve both reduction in processing time and securing necessary information.

  Various rules for determining the priority order are conceivable. Here, two criteria are exemplified.

(1) Presence / absence of vehicles in own lane In general, when driving at high speed on a highway or the like, priority should be given to a preceding vehicle running in the own lane. From this viewpoint, it is effective to increase the priority for the object area detected in the own lane. In this case, it is preferable to consider the inter-vehicle distance from the preceding vehicle in addition to the information regarding the white line (lane).

(2) Vehicle speed information In general, when driving at low speeds in alleys, shopping streets, etc., attention should be paid to people and cars popping out. From this point of view, ranging accuracy at a short distance is important. Therefore, it is effective to increase the priority with respect to a short-distance object region. Also, when driving at extremely low speeds such as when turning left or right on a narrow road or entering the garage, it is required to have a close range of accuracy around the vehicle, so it is effective to further increase the priority of the close object region. .

  According to the present embodiment, in addition to the effects of the above-described embodiments, by setting the priority order to the object region, it is possible to achieve both suppression of processing time and securing of necessary information.

1 is a configuration diagram of an image processing apparatus according to a first embodiment. Explanatory drawing of pixel block PBij set to the reference image Explanatory drawing which shows an example of a subpixel calculation method Illustration of distribution of city block distance on image plane Process explanatory diagram in image processing Timing chart of frame processing according to second embodiment Explanatory drawing of estimation of the object area | region which concerns on 3rd Embodiment Diagram showing the situation where there are multiple objects

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Stereo matching part 3 Sub pixel processing part 4 Object area | region identification part

Claims (5)

In an image processing apparatus that performs stereo matching for each partial region set on an image plane using a pair of images,
Each partial area in one image is used as a correlation source, a correlation destination of the partial area is searched in the other image, and a deviation amount of the correlation destination with respect to the correlation source is represented as a pixel value represented by an integral multiple of a pixel. A stereo matching unit to calculate,
An object region specifying unit for specifying an object region where an object exists on the image plane;
A sub-pixel processing unit that calculates a sub-pixel value having a resolution higher than that of the pixel value, limited to the partial region that corresponds in position to the object region specified by the object region specifying unit. An image processing apparatus.   The object region specifying unit is a pixel whose values are close enough to be regarded as the same object on a two-dimensional plane defined by the set of pixel values calculated by the stereo matching unit, and are adjacent in position. The image processing apparatus according to claim 1, wherein the object region is specified by grouping values.   The image processing apparatus according to claim 1, wherein the object area specifying unit specifies an area that matches a luminance pattern as an object on the luminance plane of the image as the object area.   2. The subpixel processing unit sets a region for calculating the subpixel value in a current frame based on the object region in a previous frame specified by the object region specifying unit. 4. The image processing device according to any one of items 1 to 3.   When a plurality of the object areas are detected, the sub-pixel processing unit sets a priority order for each of the recording object areas based on a preset priority order rule, and the sub-pixel processing unit sets the priority order according to the set priority order. The image processing apparatus according to claim 1, wherein a pixel value is calculated.

Images