JP2013051736A - Vehicle periphery supervision device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a vehicle periphery supervision device having a function capable of performing pitching correction even in a monocular camera without a pitch detector.SOLUTION: The vehicle periphery supervision device acquires an image imaged by a camera mounted on a vehicle at each predetermined control period, calculates displacement magnitude of the imaged image at the present control period from the previous control period, and using the displacement magnitude, calculates a pitching correction value to cancel the displacement of the imaged image caused by a vehicle behavior, generates a corrected image by correcting the present imaged image by the pitching correction value, and detects an object from the corrected image. From the imaged image, by cutting out, as a supervision area, a predetermined area upper than the horizon, a predetermined area between lower and upper than the horizon including the horizon, or a predetermined area from the self-vehicle to and inclusive of a preceding vehicle within a predetermined distance, the displacement magnitude is obtained between the supervision area of the imaged image at the present control period and the supervision area of the imaged image at the previous control period.

Description

  The present invention relates to a vehicle periphery monitoring device that detects a monitoring object around a vehicle from an image captured by a camera provided in the vehicle.

  When monitoring the surroundings of a vehicle while traveling using a camera provided on the vehicle, even if the visual axis of the camera is displaced due to pitching (vertical vehicle shake) or the like, It is desirable that it can be detected.

  Accordingly, an environment recognition device for a mobile vehicle is proposed in which a pitch detector such as a gyro sensor is provided in a vehicle and an image shift amount due to pitching is corrected based on a vertical displacement amount detected by the pitch detector. (For example, refer to Patent Document 1).

  Also, based on the parallax of the image captured by the stereo camera provided in the vehicle, the distance between the host vehicle and the object is calculated, and the relative speed and relative adjustment between the host vehicle and the object are calculated from the change of the distance for each sampling period. There has been proposed a vehicle distance measuring device that calculates a speed and determines that a pitching change of a predetermined amount or more has occurred (see, for example, Patent Document 2). In this vehicle distance measuring device, the correction amount of the image position corresponding to the degree of pitching is calculated based on the relative acceleration between the host vehicle and the object.

Japanese Patent Laid-Open No. 3-118611 Japanese Patent Laid-Open No. 11-51645

  In order to perform pitching correction with the configuration described in Patent Document 1 described above, a pitch detector is required, which increases the cost of the device and complicates the configuration of the device. Furthermore, there is a possibility that the pitching correction becomes inappropriate due to a shift in timing for correcting the displacement amount detected by the pitch detector in the captured image.

  In addition, the configuration described in Patent Document 2 is for measuring the distance from an object with a stereo camera, and thus has a disadvantage that it cannot be applied to a configuration with a monocular camera.

  Accordingly, an object of the present invention is to provide a vehicle periphery monitoring device having a function capable of performing pitching correction even in a configuration using a monocular camera without using a pitch detector.

  The vehicle periphery monitoring device according to the first aspect of the present invention includes an image acquisition unit that acquires a captured image by a camera mounted on a vehicle every predetermined control period, and a displacement of the captured image in the current control period from the previous control period. A pitching correction value calculating unit for calculating a pitching correction value for calculating a pitching correction value for offsetting a displacement of the captured image caused by the behavior of the vehicle using the displacement amount, and for the current captured image, the pitching A correction image generation unit that generates a correction image corrected by a correction value; and an object detection unit that detects an object from the correction image, wherein the pitching correction value calculation unit is above the horizon from the captured image. A predetermined region including the horizon, including a horizon, from a lower side to an upper side of the horizon, or a predetermined region including a preceding vehicle within a predetermined distance from the host vehicle as a monitoring region. And obtaining the amount of displacement between the monitoring region of the captured image in the monitor area and the control period of the previous.

  According to the first aspect of the invention, pitching correction can be performed even in a configuration using a monocular camera without using a pitch detector.

  In the vehicle periphery monitoring device according to a second aspect of the present invention, in the first aspect, the pitching correction value calculating unit moves up and down in a monitoring region of the captured image in the previous control cycle and a monitoring region of the captured image in the current control cycle. The displacement amount is calculated based on the correlation value of the relative displacement amount of the image portion of the object not to be processed.

The block diagram of a vehicle periphery monitoring apparatus. The flowchart of a vehicle periphery monitoring process. Explanatory drawing of correction | amendment of the captured image by a pitching correction value. The flowchart of a pitching correction value calculation process. Explanatory drawing of the setting aspect of a monitoring area. Explanatory drawing of rearrangement of the coordinate system of the displacement amount with respect to the inverse Fourier transform map of a phase correlation function. Explanatory drawing of the effect by using the improved phase correlation function. Verification data when improved phase correlation function is used. Explanatory drawing of the effect by using another improved phase correlation function.

  An embodiment of the present invention will be described with reference to FIGS.

  With reference to FIG. 1, the vehicle periphery monitoring device of the present embodiment is used by being mounted on a vehicle, and an ECU (Electronic Control Unit) 10 and an infrared camera 20 connected to the ECU 10 (corresponding to the camera of the present invention). , A yaw rate sensor 21, a vehicle speed sensor 22, a display 30, a speaker 31, and an actuator 32.

  The ECU 10 is an electronic unit configured by a CPU, a memory, and the like (not shown). By causing the CPU to execute a vehicle periphery monitoring program held in the memory, the ECU 10 captures an image captured by the infrared camera 20 every predetermined control period. A pitching correction value calculation unit 12 that calculates a pitching correction value PY for canceling the displacement amount due to the vehicle behavior for the captured image f in the current control cycle acquired by the image acquisition unit 11. A corrected image generating unit 13 that generates a corrected image f3 obtained by correcting the captured image f with the pitching correction value PY, an object detecting unit 14 that detects an object (such as a pedestrian) from the corrected image f3, and an object that is an alarm target. Alarm control unit 15 that gives an alarm by display 30 and speaker 31 when detected, and contact with object The actuator 32 (brake mechanism, steering mechanism, etc.) in order to avoid serving as a vehicle control unit 16 to operate the.

  Next, the execution procedure of the vehicle periphery monitoring process by the ECU 10 will be described according to the flowchart shown in FIG. The ECU 10 starts the vehicle periphery start process according to the flowchart shown in FIG. 2 when the driver performs an operation for starting the vehicle periphery monitoring process.

  After performing the initialization process in STEP 1, the ECU 10 repeatedly executes the processes of STEP 2 to STEP 9 at a predetermined control cycle until the driver performs a stop operation of the vehicle periphery monitoring process in STEP 10. STEP 2 is a process performed by the image acquisition unit 11. The image acquisition unit 11 converts an analog video signal for one frame output from the infrared camera 20 into a digital signal and holds it in an image memory (not shown).

  Subsequent STEP 3 to STEP 6 are processes performed by the pitching correction value calculation unit 12 and the corrected image generation unit 13. The pitching correction value calculation unit 12 determines that the vehicle behavior (step 5) is determined in STEP 3 when it is determined in STEP 3 that the vehicle is not stopped from the speed detected by the vehicle speed sensor 22 and the yaw rate sensor 21 determines that the vehicle is not turning in STEP 4. A pitching correction value PY for correcting the displacement of the captured image due to pitching or the like is calculated. The pitching correction value PY is a shift amount (pixel value) for canceling out the displacement of the captured image.

  Further, the correction image generation unit 13 determines that the vehicle is running from the speed detected by the vehicle speed sensor 22 in STEP 3, and determines that the vehicle is not turning by the yaw rate sensor 21 in STEP 4, and calculates the pitching correction value in STEP 5. When the pitching correction value PY is calculated by the unit 12, the current captured image f1 is shifted by the pitching correction value PY to generate a corrected image f3, and the process proceeds to STEP7.

  Thus, for example, as shown in FIG. 3, when the captured image f in the current control cycle is displaced upward due to pitching, the corrected image f3 is shifted downward by the pitching correction value PY. Is generated. In this case, since the upper side of the pedestrian image portion 50 that has deviated from the object search area E in the captured image f falls within the search area E in the corrected image f3, the pedestrian image portion 50 is displayed from the corrected image f3. Can be detected.

  Further, when it is determined in STEP 3 that the vehicle is stopped and when it is determined in STEP 4 that the vehicle is turning, the process branches to STEP 10. In this case, the corrected image f3 is not generated, and an alarm and contact avoidance control by STEP7 to STEP9 described later are not executed.

  STEP 7 is processing by the object detection unit 14. The object detection unit 14 starts from the corrected image f 3 when the corrected image f 3 is generated in STEP 6, and from the captured image f when the corrected image f 3 is not generated. Detecting an object to be monitored (such as a pedestrian).

  Subsequent STEP 8 to STEP 9 are processes by the alarm control unit 15 and the vehicle control unit 16. The alarm control unit 15 determines whether or not the object detected by the object detection unit 14 in STEP 8 is an alarm object (determined based on the distance from the own vehicle, the traveling direction of the object, etc.), and the alarm When it is determined that it is the target, the process proceeds to STEP9.

  Then, the alarm control unit 15 displays a warning on the display 30 by indicating the position of the target object, and outputs a sound for notifying the presence of the target object from the speaker 31. Further, the vehicle control unit 16 activates an actuator (brake mechanism, steering mechanism, etc.) as necessary in order to avoid contact with an object that is an alarm target.

  On the other hand, when it is determined in STEP8 that the object detected by the object detection unit 14 is not an alarm target, the process branches to STEP10, and the process of STEP9 by the alarm control unit 15 and the vehicle control unit 16 is not performed.

  Next, the pitching correction value setting process by the pitching correction value calculation unit 12 will be described with reference to the flowchart shown in FIG.

  In STEP 20, the pitching correction value calculation unit 12 cuts out the monitoring region W from the captured image f in the current control cycle. Here, the monitoring area W is set at the center of the captured image f as shown in FIG.

  From the viewpoint of including an image of an object that does not move up and down in the monitoring area, as shown in FIG. 5B, the monitoring area W is placed on the sky side (above the horizon H, usually a building or The preceding vehicle may be imaged). Further, from the viewpoint that the amount of movement on the image corresponding to the movement of the host vehicle in the front-rear direction is small, the monitoring area W is changed to an area where the vicinity of the horizon H is monitored more extensively as shown in FIG. It may be set.

  Alternatively, when an image portion of a preceding vehicle close to the host vehicle is detected from the captured image in the previous control cycle, the preceding vehicle is an object with a small vertical displacement on the captured image, so this image portion is The monitoring area may be set so as to include it.

  In the subsequent STEP 21, a two-dimensional operator such as a Hamming window is applied to the monitoring area. Here, the reason why the two-dimensional operator is applied to the monitoring region is to prevent the boundary of the monitoring region from affecting the result of the fast Fourier transform described later. In STEP 22, the pitching correction value calculation unit 12 performs a fast Fourier transform on the monitoring area of the captured image f in the current control cycle to generate a Fourier transform image F.

  At the next STEP 23, the pitching correction value calculation unit 12 determines that the captured image f is an initial image (the first captured image after the processing according to the flowchart of FIG. 2 is started and the vehicle is stopped at STEPs 3 and 4 of FIG. It is determined whether or not it is the first captured image after it is determined that the vehicle is turning. When the captured image f is an initial image, the process branches to STEP 40, and the pitching correction value calculation unit 12 stores the Fourier transform image F in the memory as F1 (Fourier transform image in the previous control cycle). In subsequent STEP 41, the pitching correction value calculation unit 12 clears the count value FR of the frame counter (0 → FR), sets the pitching correction value PY to zero in STEP 42, proceeds to STEP 29, and ends the process.

  On the other hand, when the captured image f is not the initial image in STEP 23, the process proceeds to STEP 24, and the pitching correction value calculation unit 12 holds the Fourier transform image F in the memory as F 2 (Fourier transform image in the current control cycle). Then, in subsequent STEP 25, the pitching correction value calculation unit 12 obtains the inverse Fourier transform IRa of the phase correlation function Ra (ω) of the following equation (1) and obtains the current captured image with respect to the captured image in the previous control cycle. The amount of displacement is obtained.

  However, F1: Fourier transform image in the previous control cycle, F2: Fourier transform image in the current control cycle, ε1: Afterimage portion of the captured image in the previous control cycle included in the captured image in the current control cycle Constant to reduce the effect of.

  The inverse Fourier transform IRa of the phase correlation function Ra (ω) according to the above equation (1) is a two-dimensional map of complex elements having the same size as the image cut out as the monitoring region. When the positions of the elements of the two-dimensional map are switched as shown in FIGS. 6A and 6B, the horizontal position is larger than M / 2 and the vertical position is larger than N / 2. Is indicated by a negative coordinate value.

  When the horizontal position ix and the vertical position iy of the element having the maximum real part are obtained in the IRa map in which the coordinates are set in this way, the horizontal position ix and the vertical position iy are respectively the displacement amounts of the captured image in pixel units. The horizontal and vertical components of It should be noted that by using the values of elements in the vicinity of (ix, iy) of the map of IRa, the position of the sub-pixel unit at which the real part is maximum when IRa is a continuous system instead of a discrete system is estimated, and this position is calculated. The horizontal and vertical components of the displacement amount of the captured image may be used.

  Here, the infrared imaging element used in the infrared camera 20 is configured using an infrared absorption film that converts incident infrared rays into heat and absorbs them, and changes in electrical resistance in accordance with the amount of absorbed heat. . Then, it takes time depending on the thermal time constant τ determined by the heat capacity C of the infrared absorption film and the thermal conductance G of the surrounding circuit until the heat once absorbed by the infrared absorption film is released.

  Therefore, there may be a phenomenon in which the captured image in the previous control cycle overlaps the captured image in the previous control cycle as an afterimage due to a delay in response of the infrared detection signal to the change in the level of the infrared light input to the infrared absorption film. .

  Here, FIG. 7A shows a general phase correlation function R (ω) of the following formula (2), and shows a map of the inverse fast Fourier transform IR, with the base of the two-dimensional complex element. This is a three-dimensional display with (xy) coordinates and the height as the size of the real part of the complex element, and there are two (P1, P2) plural elements (peak points) with the maximum real part. doing.

  In this case, one of the two peak points corresponds to the displacement of the captured image due to the behavior of the vehicle, and the other corresponds to the afterimage of the captured image in the previous control cycle. It cannot be distinguished from the size of the real part which one of the two corresponds to the displacement of the captured image due to the behavior of the vehicle.

  Therefore, in the present embodiment, Ra (ω) in the above equation (1) is obtained instead of R (ω) in the above equation (2). In the above equation (1), ε1 is a constant for reducing the influence of the remaining amount caused by the influence of the heat capacity of the infrared imaging element of the infrared camera 20, and is determined by experiment or computer simulation. Since the degree of the influence of the afterimage depends on the characteristics of the infrared camera 20, once an appropriate ε1 is set, it does not need to be changed later.

  FIG. 7B shows the same equation (1) for F1 (Fast Fourier transform image in the previous control cycle) and F2 (Fast Fourier transform image in the current control cycle) as in FIG. The map of the inverse Fourier transform IRa of Ra (ω) is shown in the same manner as in FIG. 7A, and the peak point P2 seen in FIG. 7A has disappeared. Therefore, the amount of deviation of the remaining peak point P1 from the map center can be obtained as the amount of displacement from the captured image in the previous control cycle of the captured image in the current control cycle.

  In the next STEP 26, the pitching correction value calculation unit 12 updates the Fourier transform image F1 of the previous control cycle with the Fourier transform image F2 of the current control cycle. In STEP 27, the frame counter is counted up (FR + 1 → FR), and an attenuation coefficient α for preventing an error from accumulating in the pitching correction value PY is calculated by the following equation (3).

  Where α is the attenuation coefficient, D is the maximum value of α (the value of α after the period indicated by FX), FR is the count value of the frame counter, and FX is the period during which the pitching correction value PY is suppressed (immediately after turning, etc.) The number of frames corresponding to the period during which the vehicle posture becomes unstable.

  In subsequent STEP 28, the pitching correction value calculation unit 12 calculates (updates) the pitching correction value PY by the following equation (4). Then, proceeding to STEP 29, the pitching correction value calculation unit 12 ends the process.

  PY: pitching correction value, SY: image displacement amount, α: attenuation coefficient.

  Here, FIG. 8 shows an actual pitching amount (a in the figure), a pitching correction value (b in the figure) calculated using the phase correlation function R (ω) of the above equation (2), and the above equation (1). The pitching correction value (c in the figure) calculated using the phase correlation function Ra (ω) of) is shown with the vertical axis set to the pitching amount (number of pixels) and the horizontal axis set to the frame number of the captured image. It is a comparative graph. As is apparent from FIG. 8, the difference between the pitching correction value and the actual pitching amount can be reduced by using the phase correlation function Ra (ω) to which the constant term ε1 of the above equation (1) is added.

  Instead of the phase correlation function Fa (ω) in the above formula (1), a phase correlation function Fb (ω) in the following formula (5) may be used.

  However, F1: Fourier transform image in the previous control cycle, F2: Fourier transform image in the current control cycle, ε2: Afterimage portion of the captured image in the previous control cycle included in the captured image in the current control cycle Constant to reduce the effect of.

  Here, FIG. 9 (a) shows a map of the inverse fast Fourier transform IR of the phase correlation function R (ω) of the above equation (2), and the bottom surface of the complex element 2 as in FIG. 7 (a). 3D display with dimension (xy) coordinates and height as the size of the real part of the complex number element, two multiple elements (peak points) with the maximum real part (P3, P4) Existing.

  On the other hand, FIG. 9B shows the above formulas for F1 (Fast Fourier transform image in the previous control cycle) and F2 (Fast Fourier transform image in the current control cycle) as in FIG. 9A. The phase correlation function Rb (ω) of (4) is obtained, and the map of the inverse fast Fourier transform IRb is shown in the same manner as in FIG. 9A.

  In FIG. 9 (b), the size of the real part of P4 is reduced to one peak point (P5). Therefore, the pitching correction value calculation unit 12 can obtain the amount of deviation from the center (0, 0) of P5 as the amount of displacement from the captured image in the control cycle of the captured image in the current control cycle. .

  In the present embodiment, it has been described that the phase correlation function of the above formula (1) or the above formula (5) is used in order to suppress the influence of the afterimage when the infrared camera 20 is used. In general cameras including the above, the phase correlation function of the above formula (1) or the above formula (5) is also used to suppress the influence of the image due to the fixed noise always generated at a specific position (fixed position) of the image sensor. Is effective.

  The present invention is also applicable to the case where the displacement amount of the captured image is obtained using the general phase correlation function of the above equation (2) without using the phase correlation function of the above equations (1) and (5). The effect of can be obtained. In this case, as shown in FIGS. 7A and 9A, when there are a plurality of peak points, the peak point farthest from the center of the inverse fast Fourier transform map (in FIG. 7A) The position of the complex element of P1, FIG. 9 (a) P2) may be the amount of displacement of the captured image in the current control cycle from the captured image in the previous control cycle.

  In the present embodiment, the displacement of the captured image caused by the vehicle behavior is corrected in two directions, x (left and right) and y (up and down), but the phase correlation in the up and down direction is obtained and only in the up and down direction. You may make it correct | amend.

DESCRIPTION OF SYMBOLS 10 ... ECU, 11 ... Image acquisition part, 12 ... Displacement amount calculation part, 13 ... Correction | amendment image generation part, 14 ... Object detection part, 20 ... Infrared camera.

Claims (2)

An image acquisition unit that acquires a captured image by a camera mounted on the vehicle for each predetermined control period;
Pitching that calculates the amount of displacement of the captured image in the current control cycle from the previous control cycle, and calculates the pitching correction value for offsetting the displacement of the captured image due to the behavior of the vehicle using the amount of displacement A correction value calculation unit;
A corrected image generation unit that generates a corrected image obtained by performing correction using the pitching correction value for the current captured image;
An object detection unit for detecting an object from the corrected image,
The pitching correction value calculation unit monitors a predetermined region above the horizon, a predetermined region including the horizon from the lower side to the upper side, or a predetermined region including a preceding vehicle within a predetermined distance from the host vehicle, from the captured image. The vehicle periphery monitoring device is characterized in that the displacement amount is obtained between the monitoring region of the captured image in the current control cycle and the monitoring region of the captured image in the previous control cycle. The vehicle periphery monitoring device according to claim 1,
The pitching correction value calculation unit is based on the correlation value of the relative displacement amount of the image portion of the object that does not move up and down in the monitoring region of the captured image in the previous control cycle and the monitoring region of the captured image in the current control cycle. The vehicle periphery monitoring device characterized in that the displacement amount is calculated.