JP2007234019A - Vehicle image area specifying device and method for it - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a vehicle image area specifying device and a method for it specifying an image area to include a vehicle included in an image even if the vehicle is behind a projection shadow of another object, if uneven illumination light is radiated, or if there is no lane. <P>SOLUTION: This vehicle image area specifying device based on a vehicle shadow is provided with an image input means inputting an image photographing a traveling condition of the vehicle on a road, a grade distribution detection means calculating a grade distribution on the image inputted by the image input means, a vehicle shadow candidate area specification means specifying a candidate area for the vehicle shadow based on grade information of the image, and an image area specification means specifying an image area ROI including the vehicle image based on the vehicle shadow candidate area. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a vehicle image region specifying device and method for specifying an image region including a vehicle image, and in particular, an image including a vehicle image based on the vehicle shadow based on the detection of a vehicle shadow based on a luminance gradient. The present invention relates to a vehicle image area specifying apparatus and method for specifying an area.

  In the field of Intelligent Transport System (ITS), when a vehicle image is identified by machine vision, it is first necessary to cut out an image of a region where the vehicle exists from the captured image. Usually, based on the characteristics of the vehicle to be identified or the similarity of the characteristics, an area where the vehicle can exist is identified from the captured image, and the image is cut out from the area. An image region in which such a vehicle can exist is abbreviated as a region of interest ROI (Region of Interest). FIG. 1 shows an example of a region of interest ROI including a vehicle image CIM.

  The presence of shadows under the vehicle due to irradiation with sunlight or other light sources is a common feature of vehicles with different shapes, so the vehicle shadow is a vehicle image in vehicle image generation (Machine Vision) technology. This is the main feature that is used as the basis for taking out. The prior art discloses a vehicle image area specifying method based on the vehicle shadow feature as described below. The first vehicle image region specifying method based on the underside of the vehicle specifies the vehicle image region from the image based on the feature that the luminance of the underside of the vehicle is darker than the road surface. As shown in FIG. 2, in this method, in step S1, an image of the road surface condition (frontal landscape) captured by the imaging device is input, and in step S2, the road surface luminance average is calculated for the road surface condition image input in step S1. The value is calculated and this average value is set as a reference value. In step S3, the brightness value of each pixel in the image is compared with the reference value, and all areas whose brightness values are lower than the reference value are possible. The candidate area is identified as a candidate shadow area, that is, the candidate area of the vehicle shadow, and then the candidate area is divided according to the distance between the vehicle and the candidate area, the size and angle of the candidate area, and finally In step S4, the result after the allocation in step S3, that is, the image region ROI including the vehicle image is output.

However, as shown in FIGS. 3 and 4, when the vehicle itself is within the projection shade of another three-dimensional object (for example, the building BL shown in the figure), the projection shade is also darker than the road surface. If the area of the vehicle image is specified using the average luminance value as a criterion for judgment, it becomes an erroneous judgment that the projection shade of the building BL is also the shade under the vehicle. As shown in FIG. 3, the image area CMA captured by the imaging device is much larger than the vehicle area, and therefore this vehicle area cannot be accurately specified as the ROI. In addition, as shown in FIG. 4, if the image areas ROI1 and ROI2 are specified, the entire vehicle image is not included in the image areas ROI1 and ROI2, so that the vehicle cannot be identified.
By the shade formation principle, it can be seen that the shadow area under the vehicle is always darker than the surrounding area due to the luminance superposition. That is, the area under the vehicle shadow is always darker than the surrounding area even in the shadow of the projection or in different illumination. The conventional method of identifying the area under the vehicle shade according to the average brightness value does not consider this feature of the vehicle shade, so that the vehicle is in the shadow of another object or uneven illumination. In some cases, there was a problem that the vehicle area could not be specified accurately.

Prior art for solving the problem that the vehicle area cannot be accurately identified in consideration of the fact that the shadow area under the vehicle always has a darker characteristic than the surrounding area, and when the vehicle is in the shadow of another object (See Patent Document 1). This prior art vehicle image area specifying technique is shown in FIG. First, in step S21, an image of the road surface condition (see FIG. 6) captured by the imaging device is input. In step S22, a lane identification line (white line) 31 of the traveling lane of the host vehicle is detected. In step S23, a luminance value distribution on the road surface of the traveling lane 32 of the host vehicle is calculated from the detected lane identification line, and a luminance value distribution function D is obtained from the luminance value distribution. As shown in FIG. 6, the luminance value distribution is the average of the road surface luminance on the traveling lane 32 at an arbitrary Y coordinate value when the lowest position of the traveling lane is Y = 0 and the front of the traveling lane is the Y axis. This is a value and does not include a shadow. In other words, the luminance distribution of the traveling lane 32 for each Y coordinate is obtained by obtaining the average value of the luminance values of the pixels on the traveling lane 32 having the same Y coordinate. When the driving lane 32 does not include shadows or preceding vehicles, it is known that the upper brightness in the front image (distant from the road, Y coordinate value is larger) indicates a brighter luminance value, and the luminance value is the Y coordinate value. Is approximately proportional to For this reason, for example, only the luminance value distribution from the lowest point (Y = 0) to the predetermined value (Y = Y ₀ ) of the front image is obtained, and the luminance distribution linear approximation formula (Y The luminance value distribution can be obtained by obtaining the luminance value distribution function. The luminance value distribution function D is expressed by D = αY + β, and α and β are identified from the luminance value distribution in the region where the Y coordinate value is ₀ to Y0.

In step S24, the luminance values of all the pixels in the traveling lane of the host vehicle are compared with the setting values obtained from the luminance value distribution function, and all areas where the luminance values are lower than the setting values are determined. Set as a candidate area under the vehicle shadow. In step S25, based on the feature that the vehicle shade behind the other object's projection shade is darker than the projection shadow of the other object, the brightness value distribution in the candidate shade of the vehicle shade causes the Detect dark areas, identify these areas as shadow areas under the vehicle, and finally, in step S26, output a predetermined image area ROI including a vehicle image based on the shadow area under the vehicle identified in step S25. To do.
JP 2003-76987 A

However, in the conventional method, first, the lane identification line (white line) 31 of the traveling lane of the own vehicle must be detected, and the luminance value distribution on the road surface of the traveling lane must be obtained based on the lane identification line. For this reason, when there is no lane identification line on the road surface, or when the lane identification line is not imaged in the captured image, the traveling lane surface for obtaining the luminance value distribution cannot be determined, and further steps are performed. This makes it impossible to complete the processing for specifying the image area ROI including the vehicle image.
Also, as shown in FIG. 7, under natural conditions, the road surface brightness is affected by illumination, projections of other buildings, and changes in road surface color, and the road surface brightness distribution is complex and irregular. In the prior art method, since the luminance value distribution on the road surface in the traveling lane of the own vehicle is calculated, this luminance value distribution is a complicated and irregular luminance distribution state of a multi-lane road surface such as a two-lane or three-lane road. Cannot be represented and cannot be used in the case of multiple lanes.
In the prior art method, the region for calculating the luminance value distribution on the road surface is the region from the lowest part (Y = 0) to the upper part (Y = 0 to Y ₀ ) of the image. When there is a dark region (see FIG. 8), the threshold value calculated from the luminance value distribution function D obtained from the luminance value distribution on the road surface may be too small. For this reason, when a pixel having a luminance smaller than the threshold value is set as a pixel in the shadow area under the vehicle, the image area ROI including the vehicle image cannot be accurately specified.
As described above, the present invention includes an image including all vehicles included in the image even when the vehicle is in the shadow of another object, even when there is uneven illumination, or when there is no lane. It is an object of the present invention to provide a vehicle image area specifying device and method for accurately specifying an area ROI.
It is another object of the present invention to provide a vehicle image area specifying apparatus and method that can be widely used even on road surfaces on multiple roads.

-Vehicle image area specifying method A first aspect of the present invention is a vehicle image area specifying method for specifying an image area including a vehicle image, and an image input step for inputting an image obtained by imaging a running situation of a vehicle on a road; A gradient distribution detection step for calculating a gradient distribution for the image input in the image input step, and a vehicle under-shadow candidate region specifying step for specifying a vehicle under-shadow candidate region based on the gradient information of the image, A step of identifying an image region ROI including a vehicle image based on the candidate area under the vehicle shadow.
In the vehicle image area specifying method of the present invention, the candidate area for the vehicle shadow is specified based on the fact that the area under the vehicle shadow is darker than the surrounding area in the candidate area specification for the vehicle shadow. It is characterized by doing.
According to the vehicle image region specifying method of the present invention, the difference value in the vertical direction from the image is negative based on the gradient distribution information of the image detected in the gradient distribution detecting step of the image in the candidate region specifying step under the vehicle shadow. The method includes a step of extracting all pixel points that are values.
The vehicle shadow region candidate region specifying step of the vehicle image region specifying method of the present invention includes the following two conditions for the extracted pixel point: (a): the gradient value of the pixel point of interest is the gradient direction (B): Does the two adjacent points described in (a) satisfy that they are pixel points adjacent in the vertical direction? And determining a pixel point satisfying these two conditions as an edge point of a candidate area under the vehicle.

-Vehicle image area specifying device A second aspect of the present invention is a vehicle image area specifying device for specifying an image area including a vehicle image, and an image input means for inputting an image obtained by imaging a running situation of a vehicle on a road. A gradient distribution detection means for calculating a gradient distribution for the image input by the image input means; a vehicle shadow candidate area specifying means for specifying a vehicle shadow area based on the gradient information of the image; Means for specifying an image area ROI including a vehicle image based on the candidate area under the vehicle shadow.
Further, the vehicle image area specifying device of the present invention realizes each step of the above-described vehicle image area specifying method and specifies the image area ROI including the vehicle image.

According to the present invention, it is possible to accurately specify a region including a vehicle image even when the vehicle is in the shadow of another object, unevenly illuminated, or when there is no lane.
Moreover, according to this invention, it can be spread and used also for the road surface condition of a multilane.

  FIG. 9 is a structural diagram of a vehicle image area specifying device according to an embodiment of the present invention. As shown in FIG. 9, the vehicle image area specifying device of the present embodiment includes an image input means 11 such as a television camera and a surveillance image pickup device, a gradient distribution detecting means 12, and a candidate area specifying means 13 under the vehicle. The image area specifying means (ROI specifying means) 14 including the vehicle image is included. The gradient distribution detecting means 12, the vehicle shadow area candidate specifying means 13, and the image area specifying means (ROI specifying means) 14 constitute an image processing unit. Although not shown, a display unit such as a liquid crystal is connected to the image processing unit so that an image input from the image input unit 11 or a predetermined image is displayed.

Hereinafter, the image region specifying processing method of the vehicle image region specifying device of the present embodiment will be described with reference to the processing flows of FIGS. 10 and 11.
The image input means 11 mounted on the vehicle inputs an image (an image showing a vehicle traveling situation ahead of the road) captured by an imaging device such as a television camera or a surveillance imaging device to the gradient distribution detection means 12 (step S11). .
The gradient distribution detecting means 12 calculates the luminance gradient distribution at all pixel positions (x, y) of the image input by the image input means 11 using equation (1) (step S12). FIG. 12 shows the XY coordinate system (screen coordinate system) of the image IMG, with the lower left corner as the origin.

(1)式において、f(x，y)は位置(x，y)の画素P(x,y)の輝度を示す輝度関数であり、輝度勾配G(x，y)は輝度関数f(x，y)の各軸方向の一次微分係数で示され、方向及び振幅（大きさ）を有するベクトルである。
輝度勾配の方向は、画素位置(x，y)における最大の変化率の方向であり、(2)式で示される。

勾配の振幅（大きさ）は、画素位置(x，y)における勾配の各軸方向の変化強度であり、次式で示される。

In equation (1), f (x, y) is a luminance function indicating the luminance of the pixel P (x, y) at the position (x, y), and the luminance gradient G (x, y) is the luminance function f (x , Y) is a vector having a direction and an amplitude (magnitude) indicated by a first-order differential coefficient in each axial direction.
The direction of the luminance gradient is the direction of the maximum rate of change at the pixel position (x, y), and is expressed by equation (2).

The amplitude (magnitude) of the gradient is a change intensity in each axial direction of the gradient at the pixel position (x, y), and is represented by the following equation.

実際は(1)式の計算を簡単に行うために、差分値Gx，Gyを(4)式により近似計算する。

ステップＳ１２において、(4)式により画像の所定画素の水平方向及び垂直方向での輝度の差分を算出した後、(2)〜(3)式を用いて画素の勾配を算出する。そして全画素について上記計算を行って画像全体の勾配分布を算出する。

Actually, in order to easily calculate the equation (1), the difference values Gx and Gy are approximated by the equation (4).

In step S12, after calculating the luminance difference in the horizontal direction and the vertical direction of the predetermined pixel of the image by the equation (4), the gradient of the pixel is calculated by using the equations (2) to (3). Then, the above calculation is performed for all pixels to calculate the gradient distribution of the entire image.

The gradient distribution obtained by the calculation in step 12 includes information on relative change in luminance at all pixels of the image. That is, regardless of whether the image includes images of multiple lanes, or whether the image includes road surface conditions with multiple different illuminations, or whether the image is a projection of roadside buildings and road surfaces. Regardless of whether or not various colors are included, the gradient distribution includes information on the shade of all vehicles in the current image.
After detecting the gradient distribution of the image, the gradient distribution detection means 12 inputs the result to the candidate area specifying means 13 under the vehicle shadow.

  The vehicle shadow candidate area specifying unit 13 extracts a vehicle shadow candidate area from the image based on the gradient information input from the gradient distribution detection unit 12 (step S13). The extraction of the candidate area under the vehicle shadow uses a feature in which the luminance of the image in the vehicle shadow area is lower than the luminance of the surrounding area. Specifically, the fact that the difference value Gy in the vertical direction (Y direction) at the edge pixels in the dark shadow area under the vehicle is a negative value is used. Therefore, based on this feature, the vehicle shadow area candidate area specifying unit 13 marks all pixel points having a negative gradient value Gy in the vertical direction in step S131 shown in FIG.

Next, in step S132, the vehicle shadow candidate area specifying unit 13 determines whether or not the marked pixel points satisfy the following two predetermined conditions (a) and (b), and all the pixel points that satisfy the conditions are satisfied. Is identified as an edge candidate point under the vehicle shadow. These two conditions are
(a): the gradient value of the current pixel point is greater than the gradient values of two pixel points adjacent to this point in the gradient direction;
(b): The two adjacent points described in (a) are pixel points adjacent in the vertical direction;
It is. In order to recognize the shade of the vehicle, the feature that “the shade under the vehicle is darker than the surrounding area” is used as described above. Specifically, the point where the pixel gradient is negative in the vertical direction and the gradient has the maximum local change is marked as a point that may constitute a shade under the vehicle, and the area where the mark points are concentrated , And the area where there is a possibility of shade under the car. With reference to FIG, [i, j] of FIG. 13 (A) in the point interest, a ₀ ~a ₇ is eight pixels surrounding remarked points. (B) shows the direction of the gradient formed by the point of interest and the eight pixel points a _{0 to} a ₇ . In the present invention, two adjacent points a ₁ and a ₅ in two directions (vertical direction) of 90 degrees and 270 degrees are set as points adjacent to the point of interest [i, j] in the gradient direction. Then, the magnitudes of the gradients of the two points a ₁ and a ₅ and the point of interest are compared, and when the gradient of the point of interest is the maximum, the point of interest is marked as a point that may constitute a shadow of the vehicle To do. The above conditions (a) and (b) are defined as points that may constitute the shadow of the vehicle if the point of interest is the point where the local gradient in the preset direction (vertical direction) is the maximum. To do.
Thereafter, in step S133, the candidate area specifying means 13 under the vehicle shadow outputs the coordinates of the edge point under the vehicle shadow specified in step S132 to the ROI specifying means 14.

In step S14, the ROI specifying unit 14 specifies an image region ROI including a vehicle image based on the distance, size, and angle between the own vehicle and the candidate area under the vehicle, and the specified image region ROI. Is output. That is, for each candidate area under the vehicle shadow obtained by the above processing, the height is calculated from the width of the candidate area (vehicle underside) and the vehicle width and height ratio (aspect ratio), and the width, height An image region ROI having a length is specified. Note that the width of the image region ROI is wider than the width W of the candidate region, for example, 1.2 × W, so that the vehicle image is surely included in the image region ROI.
After that, the depth and width in the world coordinate system of all candidate ROIs are calculated by a method of verification with an imager, and the shade of the vehicle is determined based on the depth and width. That is, an ROI having a width that is too large or too small is excluded, misidentification due to noise interference is eliminated, and an accurate ROI is finally obtained and output.
That is, after a candidate area under the vehicle shadow is found, a camera calibration method is used to calculate the depth (depth) and width of the candidate area in the world coordinate system. If both the depth (depth) and the width of the candidate area in the world coordinate system satisfy the requirements of the recognition target vehicle, the candidate area can be regarded as being behind the vehicle. For example, in FIG. 14, the size in the camera coordinate system (screen coordinate system) of the image region ROI1 generated by the shadow under the vehicle and the image region ROI2 generated by the projection shadow of the roadside building is the same. However, if the calculation of the camera calibration (Calibration) calculates that the image area ROI1 is 2 meters wide in the world coordinate system and the image area ROI2 is 8 meters wide in the world coordinate system, obviously, The image area ROI1 is behind the vehicle, and the image area ROI2 exceeds the normal vehicle width and can be determined not to be under the vehicle. In addition, a candidate area whose width is considerably smaller than a normal vehicle width in the world coordinate system is determined as noise and excluded from recognition targets. Further, when the distance to the recognition target vehicle is within 50 m and the depth (depth) of the candidate area in the world coordinate system exceeds 50 m, the candidate area is excluded from the recognition target.
Finally, in step S15, the obtained ROI is output.

Although one embodiment of the present invention has been described above, various modifications may be made to the embodiment. For example, the gradient distribution detection unit 12 may approximately calculate the gradient distribution without obtaining the primary difference with the equation (4) with respect to the luminance of each pixel of the image. Using the horizontal edge operators such as Sobel operator, prewitt operator, canny operator, etc., to obtain the Gx and Gy values by processing the luminance of the pixels in the image, the equation (1) to (3) Calculate the slope value. Thus, the accuracy of extracting the edge can be improved by the function of suppressing the noise of the sobel operator and the prewitt operator.
As described above, according to the present invention, the candidate area of the vehicle shadow is specified based on the gradient distribution information of the detected image using the feature that the brightness of the image of the vehicle shadow area is always lower than the brightness of the surrounding area. Therefore, it is possible to output the image area RIO including the vehicle by specifying the vehicle shadow area more accurately than in the prior art.
Moreover, in the embodiment of the present invention, the candidate area under the vehicle shadow is specified based on the detected gradient distribution information of the image. Therefore, the present invention is not limited by the number of lanes and can be applied to the case of multiple lanes. Further, according to the present invention, not only the ROI including the vehicle image from the road surface situation image including the image ahead of the vehicle but also the ROI including the vehicle image from the road surface image including the image behind the vehicle can be specified. it can.

It is the schematic which shows a ROI area | region and the area | region of a vehicle lower shadow. It is a flowchart of an example of the vehicle image area | region identification method of a prior art. It is an example which specifies a vehicle image area | region by a prior art about the case where a vehicle is under the shadow of a building. It is another example which specifies a vehicle image area | region by a prior art about the case where a vehicle is under the shadow of a building. It is a flowchart of another example of the vehicle image area | region identification method of a prior art. It is explanatory drawing of the vehicle image area | region identification method of a prior art. It is a luminance distribution on the road surface in a normal case. This is when the lowest part of the image is dark. 1 is a structural diagram of an image area specifying device according to an embodiment of the present invention. It is a flowchart of the image area | region identification method of embodiment of invention. It is a flowchart which detects the area | region where one part is relatively dark based on the pixel gradient of embodiment of this invention. It is image coordinate system explanatory drawing. It is explanatory drawing of the determination method of a vehicle lower shade candidate point. It is explanatory drawing of the determination method of the image area ROI based on a vehicle shade.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Image input means 12 Gradient distribution detection means 13 Candidate area | region specification means 14 of a vehicle shadow 14 Image area specification means (ROI specification means) containing a vehicle image

Claims (12)

In a vehicle image region specifying method for specifying an image region including a vehicle image,
An image input step for inputting an image obtained by imaging the running state of the vehicle on the road;
A gradient distribution detection step of calculating a gradient distribution of luminance for the image input in the image input step;
A vehicle shadow area candidate step for identifying a vehicle shadow area based on the gradient information of the image;
Identifying an image area ROI including a vehicle image based on the candidate area under the vehicle shadow;
A vehicle image region specifying method comprising: In the vehicle shadow area candidate region specifying step, based on the fact that the vehicle shadow area has a darker characteristic than the surrounding area, the vehicle shadow area is specified.
The vehicle image area specifying method according to claim 1. In the vehicle shadow area candidate region specifying step, all pixel points whose difference values in the vertical direction are negative values are extracted from the image based on the gradient distribution information of the image detected in the gradient distribution detection step of the image. Including steps,
The vehicle image area specifying method according to claim 1. The candidate area specifying step of the vehicle shadow is the following two conditions for the extracted pixel point of interest:
(a): the gradient value of the pixel point of interest is larger than the gradient values of two pixel points adjacent to the point of interest in the gradient direction;
(b): the two adjacent points described in (a) are pixel points adjacent in the vertical direction;
Including determining a target pixel point that satisfies these two conditions as an edge point of a candidate area under the vehicle,
The vehicle image area specifying method according to claim 3. In the gradient distribution detection step, a luminance difference in the horizontal and vertical directions of each pixel in the image is calculated, and a gradient of each pixel is calculated based on the difference value.
The vehicle image area specifying method according to claim 1. The image input in the image input step is an image obtained by capturing a multilane road, and includes an image behind the vehicle or an image ahead of the vehicle.
The vehicle image area specifying method according to claim 1. In a vehicle image area specifying device for specifying an image area including a vehicle image,
Image input means for inputting an image obtained by imaging the driving situation of the vehicle on the road;
Gradient distribution detecting means for calculating a gradient distribution of luminance for the image input by the image input means;
A vehicle shadow area candidate means for specifying a vehicle shadow area based on the gradient information of the image;
Image area specifying means for specifying an image area ROI including a vehicle image based on the candidate area under the vehicle;
A vehicle image area specifying device comprising: The vehicle under-shadow candidate area specifying means specifies the under-vehicle shadow candidate area based on the fact that the under-vehicle shadow area has a darker characteristic than the surrounding area.
The vehicle image area specifying device according to claim 7. The vehicle shadow area candidate region specifying unit extracts all pixel points having a negative difference value in the vertical direction from the image based on the gradient distribution information of the image detected by the gradient distribution detection unit of the image. ,
The vehicle image area specifying device according to claim 7. The candidate area specifying means for the shade under the vehicle has the following two conditions for the extracted pixel point of interest:
(a): the gradient value of the pixel point of interest is greater than the gradient values of two pixel points adjacent to the point of interest in the gradient direction;
(b): The two adjacent points described in (a) are pixel points adjacent in the vertical direction;
The pixel points that satisfy these two conditions are identified as the edge points of the candidate area under the vehicle,
The vehicle image area specifying device according to claim 9. The gradient distribution detecting means calculates a luminance difference in the horizontal and vertical directions of each pixel in the image, and calculates a gradient of each pixel based on the difference value.
The vehicle image area specifying device according to claim 7. The image input by the image input means is an image of a multi-lane road, and includes an image behind the vehicle or an image ahead of the vehicle.
The vehicle image area specifying device according to claim 7.

Images