JP2011221630A - Vehicle periphery monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a vehicle periphery monitoring device that improves detection accuracy of a moving object at the periphery of a vehicle.SOLUTION: The vehicle periphery monitoring device comprises object detection means that detects a position and a velocity vector of an object at the periphery of a vehicle; imaging means that images an image at the periphery of the vehicle; multivalue processing means that applies multivalue processing to a grayscale image acquired by the imaging means; area setting means that set, on the acquired grayscale image corresponding to each of the positions of the object detected by the object detection means, an area of a predetermined size corresponding to each of the positions of the object; multivalue processing threshold value setting means that sets a threshold value for applying the multivalue processing to the corresponding grayscale image, in accordance with the size of the velocity vector of the corresponding object for each of the areas; and determination means that determines whether a predetermined target object is present or not based on the image subjected to the multivalue processing by the multivalue processing means using the corresponding multivalue processing threshold.

Description

  The present invention relates to an apparatus for monitoring the periphery of a vehicle, and more specifically to improving detection sensitivity of a moving object around the vehicle in an apparatus for monitoring the periphery of the vehicle.

  Patent document 1 discloses a pedestrian detection device. This pedestrian detection apparatus uses a laser radar and an infrared camera together, and detects a pedestrian in an infrared image based on detection information of a pedestrian candidate obtained from the laser radar.

  Patent Document 2 discloses an in-vehicle radar device. In this apparatus, in order to correct the direction of the central axis of the sensor, the captured image is divided into a plurality of regions, and vanishing points and direction vectors are obtained from the line segments detected for each region.

JP 2003-302470 A Japanese Patent No. 3862015

  When an image is captured by an infrared camera, the moving object has a lower luminance than the stationary object, that is, tends to be darker in the obtained grayscale image. This is because the imaging device of the infrared camera has a thermal time constant, so the greater the moving speed of the object relative to the infrared camera, the more the response (rise) of the detection signal (luminance signal) of the infrared camera is to the moving object than the stationary object. This is because it becomes slow.

  Therefore, when a moving object around the vehicle is detected by an infrared camera or the like, it is desired to perform the detection process in consideration of the decrease in luminance.

  However, in the inventions described in Patent Documents 1 and 2, a pedestrian or the like is not detected in consideration of the influence of the brightness reduction.

  Accordingly, the present invention improves the detection accuracy of moving objects around the vehicle by reflecting the influence of the luminance reduction of moving objects in an image acquired by an imaging means such as an infrared camera, for example, a gray scale image, in image processing. With the goal.

  The present invention provides a vehicle periphery monitoring device. The periphery monitoring device multi-values an object detection unit that detects the position and velocity vector of an object around the vehicle, an imaging unit that acquires an image around the vehicle, and a grayscale image acquired by the imaging unit. Multi-value conversion means, area setting means for setting an area of a predetermined size corresponding to each position of the object on the acquired gray scale image corresponding to the position of the object specified by the object position detection means, and for each area In addition, multi-value threshold setting means for setting a threshold value when multi-value the corresponding gray scale image according to the magnitude of the velocity vector of the corresponding object, Determining means for determining the presence or absence of a predetermined object based on the multi-valued image using the threshold value.

  According to the present invention, for each region set in the grayscale image, the multi-value threshold is set according to the magnitude of the velocity vector of the corresponding object detected by the object detection unit. The extraction accuracy of an object, particularly a moving object, can be improved. As a result, it is possible to reliably extract objects around the vehicle and promptly inform the driver.

  According to an aspect of the present invention, a vehicle periphery monitoring device is provided. The periphery monitoring device includes an imaging unit that acquires an image around the vehicle, a multi-level unit that multi-values the gray scale image acquired by the imaging unit, and the acquired gray scale image is divided into a plurality of regions. Then, a means for calculating a speed vector for each area according to the speed of the vehicle, and a threshold value for multi-leveling the corresponding gray scale image according to the size of the speed vector for each area are set. Multi-value threshold value setting means, and determination means for determining the presence or absence of a predetermined object based on the multi-value image using the corresponding multi-value threshold value by the multi-value value means. .

  According to one aspect of the present invention, the threshold value for multi-valued grayscale image is set according to the magnitude of the velocity vector calculated for each region of the grayscale image. The extraction accuracy of the target object, particularly the moving object, can be improved.

  According to one aspect of the present invention, the multi-value threshold setting unit decreases the multi-value threshold as the speed vector increases.

  According to an aspect of the present invention, it is possible to reliably extract a moving object from an acquired image even if the moving speed of the moving object around the vehicle increases.

  According to one aspect of the present invention, it further comprises means for estimating the amount of change in the acquired image caused by the behavior of the vehicle within a predetermined time, and the multi-value threshold setting means adjusts the speed vector based on the amount of change. .

  According to one aspect of the present invention, the speed vector is calculated and the multi-value threshold is set while reflecting the behavior of the vehicle, so that the accuracy of extracting an object, particularly a moving object, from the acquired image is further improved. It becomes possible to make it.

  According to one aspect of the present invention, the object detection means includes a radar, and the area setting means moves the set position of the corresponding area according to the velocity vector of the object detected by the radar.

  According to an aspect of the present invention, the object detection unit includes a radar, and the region setting unit sets the region according to the delay amount of the detection signal of the imaging unit with respect to the radar.

  According to one aspect of the present invention, the position of a region on a grayscale image is set by reflecting that a response (rise) of a detection signal (luminance signal) when detecting a moving object of an infrared camera is delayed with respect to a radar. Therefore, it is possible to more reliably extract a target object, particularly a moving object, from the acquired image.

It is a block diagram which shows the structure of the periphery monitoring apparatus of a vehicle. It is a figure which shows the processing flow in an image processing unit. It is a figure which shows the processing flow of a binarization threshold value setting. It is a figure which shows the other example of the processing flow of a binarization threshold value setting. It is a figure for demonstrating calculation of the velocity vector on an image. It is a figure which shows the processing flow in an image processing unit. It is a figure which shows the mode of the velocity vector for every area | region of an image. It is a figure for demonstrating the position shift correction in a binarized image.

  Embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a vehicle periphery monitoring device according to an embodiment of the present invention. The periphery monitoring device is mounted on a vehicle, and includes a laser radar 10, an infrared camera 12, an image processing unit 14 for detecting an object around the vehicle based on image data captured by the infrared camera 12, and a detection result thereof. And a display device for displaying an image obtained through imaging of the infrared camera 12 and for causing the driver to recognize an object around the vehicle. 18. In a vehicle provided with a navigation device, corresponding functions provided in the navigation device may be used as the speaker 16 and the display device 18.

  The image processing unit 14 in FIG. 1 has functions indicated by blocks 141 to 148 as its configuration (function). In other words, the image processing unit 14 receives a signal from the laser radar 10 and detects the position and velocity vector of an object around the vehicle, and the image around the vehicle taken by the infrared camera 12 is grayscale. An image acquisition unit 142 that acquires an image, a multi-value conversion unit 143 that multi-values the acquired gray scale image, and a gray scale image corresponding to the position of the object detected by the object detection unit 141, the position of the object A region setting means 144 for setting a region of a predetermined size corresponding to each of the threshold values, and a threshold value for multi-leveling the corresponding grayscale image for each region according to the magnitude of the velocity vector of the corresponding object The multi-value threshold setting means 145 for setting the area and the obtained gray scale image are divided into a plurality of areas, and the areas are determined according to the speed of the vehicle. Means 146 for calculating the speed vector of the vehicle, means 147 for estimating the amount of change in the acquired image caused by the behavior of the vehicle (vehicle speed, yaw rate, etc.) within a predetermined time, and a multi-value threshold corresponding to the multi-value means Functions as a determination unit 148 that determines the presence / absence of a predetermined object based on the multi-valued image.

  The “object velocity vector” used in this specification means an object velocity vector on an image obtained by projecting an object velocity vector onto an image plane unless otherwise specified, and is used in this sense throughout the entire story. ing.

  The function of each block is realized by a computer (CPU) included in the image processing unit 14. The configuration of the image processing unit 14 may be incorporated in the navigation device.

  The image processing unit 14 includes, for example, an A / D conversion circuit that converts an input analog signal into a digital signal, an image memory that stores a digitized image signal, and a central processing unit (CPU) that performs various arithmetic processes. ), A RAM (random access memory) used by the CPU to store data for calculation, a ROM (read only memory) that stores programs executed by the CPU and data to be used (including tables and maps), and driving to the speaker 16 An output circuit for outputting a signal and a display signal for the display device 18 is provided. The output signals of the laser radar 10 and the infrared camera 12 are converted into digital signals and input to the CPU.

  FIG. 2 is a processing flow executed by the image processing unit 14. This processing flow is executed at predetermined time intervals by the CPU of the image processing unit 14 calling a processing program stored in the memory. In the following description, the acquired gray scale image is used as an example of multi-value processing, and a case where a black and white image is obtained by binarization is described as an example. In the case of multi-value conversion of three or more values, the number of threshold values to be set increases, but a multi-value image can be obtained basically by performing the same processing as in the case of binarization. Moreover, although the case where a pedestrian is detected as a detection target has been described in the flow of FIG. 2, it is needless to say that the present invention is not limited to a pedestrian and can also be applied to detection of moving objects such as other animals and vehicles. .

  In step S10, a signal from the laser radar 10 is received to detect (specify) the position, size (region), and velocity vector of an object around the vehicle. Any conventional method can be used for the detection. For example, the detection is performed as follows.

  Using a scanning laser radar as the laser radar 10, the position and size (region) of an object, for example, a pedestrian candidate, is specified from the width of the detection point group. Specifically, the scanning laser radar measures the distance to the object in each direction as a detection point by receiving reflected light from the object of the beam scanned in each direction. A set of detection points is obtained by plotting the positions of these detection points in a two-dimensional coordinate system in which the position of the laser radar is the origin and the Y axis is the front direction of the radar. From these sets of detection points, detection point groups are grouped as detection point groups whose mutual distance between detection points is equal to or less than a predetermined value, and among the grouped detection point groups, pedestrian candidates whose spread width is equal to or less than a predetermined value The position and size (area) are specified.

  In the specified size (area), by defining a reference point and expressing the movement (movement amount per unit time) of the specified pedestrian candidate as a vector starting from the reference point, in that area, that is, The velocity vector for the corresponding pedestrian candidate can be obtained. The reference point can be determined by an arbitrary rule such as the center of gravity of the region, the vanishing point.

  In step S11, an output signal (that is, captured image data) of the infrared camera 12 is received as an input, A / D converted, and stored in an image memory. The stored image data is a grayscale (monochrome shade) image including luminance information. In an infrared camera in which the higher the temperature of the target, the higher the brightness is, the image data has a higher brightness value in the higher brightness portion, and the image data has a lower brightness value as the temperature becomes lower (lower brightness).

  In step S12, an area is set for the acquired grayscale image. Specifically, the area corresponding to each area (position, size) obtained in step S10 is set on the grayscale image.

  In step S13, a processing flow (routine) for setting a threshold value for binarization for each set region of the grayscale image is executed.

  FIG. 3 is a diagram showing a processing flow (routine) for binarization threshold value setting. First, step S131 or step S132 is selected. These two steps can be selected arbitrarily. In step S131 (A), the velocity vector detected by the laser radar 10 is projected onto the gray scale image area corresponding to each moving object.

  FIG. 5 is a diagram for explaining this projection. Assume that four moving bodies are detected in the scan region of the laser radar 10 indicated by reference numeral 20 as shown in the figure. Each moving body has a size (area) and a velocity vector. When the velocity vector is projected onto the gray scale image 23 for the four moving bodies, the velocity vector on the image is obtained for each region. For example, as indicated by the arrow A, the velocity vector 22 of the moving body (region) 21 is projected as the velocity vector 25 of the region 24 on the image. The same applies to other moving bodies. As described above, the speed vector projected on the image without using the speed vector detected by the laser radar as it is is projected on the image because the amount of movement on the image affects the afterimage. This is because it is more effective to use the later vector (movement amount) because the influence of the afterimage can be reflected.

  Returning to FIG. 3, when step S132 (B) is selected, a velocity vector (corresponding to the velocity vector 25 in the image 23 of FIG. 5) is used for each region on the image instead of the projection of step S131 (A). Is obtained by a conventional method such as optical flow, template matching, or phase-only correlation method. At that time, one velocity vector (movement amount) for each area may be obtained by template matching or the like, or an optical flow for each pixel is calculated and a representative value (average value, etc.) is calculated for each moving body (area). May be. Note that this step S132 (B) is a step that is executed when an object (moving body) candidate is already associated with each region on the image, and when there is no correspondence, step S131 is performed. (A) is executed.

  Next, in step S133, it is determined whether or not the object detected in step S10 in FIG. 2 is a moving body. This determination is made based on the velocity (movement) vector (size, change, etc.) for each area of the image obtained in step S131 or S132. When this determination is No, the process proceeds to step S136, and a predetermined threshold value Ths determined in advance corresponding to the stationary object is selected. The predetermined threshold Ths is stored in advance in the memory of the image processing unit 14. It should be noted that a plurality of threshold values corresponding to the magnitude of the luminance may be provided so that the threshold value can be changed according to the luminance of the background image of each region, and the threshold value can be selected from the threshold values.

  If it is determined in step S133 that the object is a moving body (Yes), in the next step S134, the size L of the velocity vector for each region, that is, the moving amount L of each moving object is calculated. In step S135, a binarization threshold value corresponding to the velocity vector magnitude (movement amount) L is calculated. For example, the binarization threshold Th is calculated as follows.

A change amount ΔTh of the binarization threshold value is determined in advance according to the magnitude (movement amount) L of the velocity vector, and the binarization threshold value Ths for the stationary object is determined as shown in the following equation (1). The binarization threshold Thm is obtained by subtracting the change amount ΔTh. The subtraction in the equation (1) is that the luminance decreases as the speed of the moving body increases. Therefore, the binarization threshold value is reduced, and detection (specification) of the moving body on the image after binarization is performed. This is to increase the accuracy.

Thm = Ths−ΔTh (1)

Alternatively, as shown in the following equation (2), a coefficient α (<1) that decreases according to the magnitude (movement amount) L of the velocity vector is determined in advance, and the binarization threshold Ths for a stationary object is set. The binarization threshold Thm may be obtained by multiplying the coefficient α (<1).

Thm = α × Ths (2)

  After obtaining the binarized threshold values (Ths, Thm) corresponding to each region, in step S14 in FIG. 2, the corresponding grayscale images are binarized using the threshold values. Specifically, a process is performed in which an area brighter than the threshold is set to “1” (white) and a dark area is set to “0” (black). By this binarization processing, an object higher than a predetermined temperature, such as a pedestrian, is extracted as a white region.

  In step S15, the object is collated from the binarized black and white image. First, it is checked whether or not it has the characteristics of the object to be excluded that is stored in the memory in advance. The excluded object means, for example, an object (a quadruped animal, a boiler, a vehicle, or the like) that is a heating element but has a clearly different shape or the like. When it does not correspond to an exclusion object, it investigates whether it corresponds to the feature of a pedestrian. The features of the pedestrian include, for example, a rounded white portion as a feature of the pedestrian's head, and an elongated white portion corresponding to the body and leg portions connected below the white portion. . Further, the similarity with a predetermined pattern representing a pedestrian stored in advance may be calculated using known pattern matching, and the similarity may be used as a material for determining whether or not the user is a pedestrian. Furthermore, since the walking speed of the pedestrian is substantially constant, information on the moving speed (speed vector) of the object obtained from the laser radar 10 may be used as one of judgment materials.

  In step S16, it is determined whether or not the determined object is a pedestrian. The determination is made based on, for example, whether or not the above-described pedestrian feature is satisfied, whether the similarity of pattern matching is high, or whether the moving speed is within a predetermined range. If it does not correspond to a pedestrian, the process ends.

  If it is determined that the person is a pedestrian, an alarm is issued in order to call the driver's attention in step S17. At that time, it is effective to change the type of alarm according to the direction in which the pedestrian is present. For example, by issuing an alarm such as “Left pedestrian attention” from the speaker 16 by voice, the driver's attention can be alerted in an appropriate direction. In the case of warning as an image, a grayscale image is displayed on the display device 18, and individual emphasis display (high brightness, blinking, color display, etc.) is performed on a recognized pedestrian in the image, Call the driver's attention.

  FIG. 4 is a diagram illustrating another example of the processing flow (routine) for setting the binarized threshold value. In FIG. 4, unlike the flow of FIG. 3, in the first step S130, it is determined whether or not the object detected in step S10 of FIG. 2 is a moving object. That is, in step S133 in FIG. 3, the presence / absence of a moving object is determined based on the speed (movement) vector on the camera image (grayscale image) acquired in step S11 in FIG. 2, but step S130 in FIG. Then, the presence / absence of a moving body is determined based on the laser radar information (movement and velocity vector of the moving body candidate) acquired in step S10 of FIG. In step S130, for example, when the magnitude of the velocity vector obtained for each object candidate is greater than or equal to a predetermined value, it is determined that the object is a moving object, and when it is smaller than the predetermined value, it is determined that the object is not a moving object.

  When this determination is No, the process proceeds to step S136, and a predetermined threshold value Ths determined in advance corresponding to the stationary object is selected. If it is determined Yes (is a moving body), the process proceeds to either step S131 (A) or step S132 (B). Since the subsequent flow including these steps is the same as the corresponding steps in FIG. 3 described above, description thereof is omitted here.

  FIG. 6 is an example of another processing flow executed by the image processing unit 14. This processing flow is executed at predetermined time intervals by calling a processing program stored in the memory by the CPU of the image processing unit 14 as in the processing flow of FIG. Moreover, it is the same as the processing flow of FIG.

  The processing flow of FIG. 6 is an example of detecting a pedestrian without assuming object detection by the laser radar 10, unlike the processing flow of FIG. In step S20, a gray scale image is acquired from the output signal of the infrared camera 12. In step S21, an area is set in the acquired gray scale image. This area setting is performed regardless of the position of the detected object. For example, nine areas divided equally as illustrated in FIG. 7 are set. The number and size of the areas are arbitrarily set according to the speed of the vehicle, the travel location (peripheral state), and the like.

  In step S22, the vehicle speed of the host vehicle is detected. Next, in order to obtain a velocity (movement) vector for each set area on the image, the process proceeds to either step S23 (A) or step S24 (B). These two steps can be selected arbitrarily. In step S23 (A), the velocity vector is obtained for each region on the image by calculating the optical flow from the temporal change of the image that changes according to the vehicle speed. FIG. 7 is a schematic diagram of velocity vectors obtained for each region. A velocity vector 25 is obtained for each region 24.

  When step S24 (B) is selected, first, in step S24, a vanishing point on the image is calculated. Any conventional method can be used as the vanishing point calculation method. Next, in step S25, a movement vector with the vanishing point on the image as a reference (starting point) is created from the vehicle speed and the acquired yaw rate. This movement vector corresponds to the amount of movement predicted according to the behavior of the host vehicle. Also in this case, a movement vector for each region as imaged in FIG. 7 can be obtained.

  Next, in step S26, it is determined whether or not a new binarization threshold value is set (calculated). When this determination is No, in step S27, the luminance of the background portion in the area is increased according to the size (movement amount) of the velocity (movement) vector corresponding to each area of the image. Specifically, the background luminance gain is increased. This is because when the host vehicle is moving, the brightness decreases as the relative speed difference from the host vehicle increases, and the brightness decreases as the brightness of the object increases. In order to increase the detection accuracy after binarization without changing (decreasing) the binarization threshold by securing the light / dark ratio (contrast) on the grayscale image. Do it.

  When a new binarization threshold value is set (calculated) in step S26, the image is obtained in step S28 by the method described with reference to FIG. 3 (FIG. 4) (selection of step S132 (B)), for example. A binarization threshold is set for each area. When the luminance adjustment in step S27 is performed, the binarization threshold value is set to a predetermined value, for example, the threshold value Ths selected in step S136 in FIG. 3 (FIG. 4). Thereafter, in steps S29 to S32, the presence or absence of a pedestrian is determined based on the binarized image by the same method as in steps S14 to S17 of FIG.

  Finally, with reference to FIG. 8, correction of misalignment in an image acquired by an infrared camera will be described. FIG. 8 is a diagram for explaining misalignment correction in a binarized image. As described above, in the infrared camera, since the rise of the luminance signal is slow in the moving body, the image is acquired later than the real image. Specifically, since the position of the real image is captured by the laser, as shown in FIG. 8A, when the object (pedestrian) is stationary, the region (target) 35 specified by the laser and the target camera image are displayed. 36 matches.

  However, when the object (pedestrian) moves, the target camera image 39 is delayed with respect to the region (target) 37 specified by the laser as shown in FIG. For this reason, in a system in which an area on a camera image corresponding to an area (target) specified by a laser is used as a detection area, there is a possibility that the object cannot be detected due to the deviation of the area. Therefore, the moving speed (vector) of the object on the camera image is obtained from the area (target) specified by the laser, and the setting position of the detection area is set to a position corrected by a delay from the real image position (shifted). ). In the example of FIG. 8, the detection area is set (corrected) by shifting from the position P (37) to the position P '(38).

  For example, the region setting method including the correction is applied to the region setting in the flow of FIGS. 2 and 6, that is, the set region is corrected later according to the magnitude of the velocity (movement) vector (the position is changed). Shift), or by setting the region by shifting the position by a predetermined correction amount from the beginning based on the magnitude of the velocity vector (movement vector) obtained last time, the detection (specification) accuracy of the object can be further improved. It becomes possible.

  The embodiment of the present invention has been described above, but the present invention is not limited to such an embodiment, and can be modified and used without departing from the spirit of the present invention. For example, the laser radar 10 may be another type of radar (for example, a millimeter wave radar), and the infrared camera 12 is also a camera that uses light (for example, visible light) in a wavelength band other than infrared (for example, visible light). A CCD camera or the like). Therefore, in the above description, the gray scale image taken by the infrared camera 12 is targeted, but the present invention is not limited to this.

10 Laser radar,
12 Infrared camera,
14 image processing unit,
16 speakers,
18 display device,
20 Laser scan range 21 Moving object,
23 Grayscale image

Claims (6)

Object detection means mounted on a vehicle for detecting the position and velocity vector of an object around the vehicle;
An imaging means mounted on a vehicle for acquiring an image around the vehicle;
Multi-value quantization means for multi-value gray scale image acquired by the imaging means;
Area setting means for setting an area of a predetermined size corresponding to each position of the object in the acquired grayscale image corresponding to the position of the object specified by the object position detection means;
Multi-value threshold setting means for setting a threshold when multi-value the corresponding grayscale image according to the size of the velocity vector of the corresponding object for each region;
Determination means for determining the presence or absence of a predetermined object based on an image multi-valued using the multi-value quantization threshold corresponding to the multi-value conversion means;
A vehicle periphery monitoring device comprising: An imaging means mounted on a vehicle for acquiring an image around the vehicle;
Multi-value quantization means for multi-value gray scale image acquired by the imaging means;
Means for dividing the acquired grayscale image into a plurality of areas and calculating a speed vector for each of the areas according to the speed of the vehicle;
Multi-value threshold setting means for setting a threshold value when multi-value the corresponding grayscale image according to the size of the velocity vector for each region;
Determination means for determining the presence or absence of a predetermined object based on an image multi-valued using the multi-value quantization threshold corresponding to the multi-value conversion means;
A vehicle periphery monitoring device comprising:   The perimeter monitoring apparatus according to claim 1, wherein the multi-value threshold value setting unit decreases the multi-value threshold value as the velocity vector increases. Means for estimating the amount of change in the acquired image caused by the behavior of the vehicle within a predetermined time;
The periphery monitoring device according to claim 1, wherein the multi-value threshold setting unit adjusts the velocity vector based on the change amount.   The periphery monitoring device according to claim 1, wherein the object detection unit includes a radar, and the region setting unit moves a set position of the region corresponding to the velocity vector of the object detected by the radar.   The periphery monitoring device according to claim 1, wherein the object detection unit includes a radar, and the region setting unit sets the region according to a delay amount of a detection signal of the imaging unit with respect to the radar.

Images