JP5416026B2 - Vehicle periphery monitoring device - Google Patents

Description

  The present invention relates to a vehicle periphery monitoring device, and more specifically to detecting and correcting a synchronization shift (phase shift) between a laser and a camera in a vehicle periphery monitoring device.

  Patent document 1 discloses a pedestrian detection device. This pedestrian detection device uses a scanning laser radar and an infrared camera together, and detects a pedestrian in an infrared image based on detection information of a pedestrian candidate obtained from the laser radar.

JP 2003-302470 A

  However, if there is a synchronization shift (phase shift) between the output signals of the laser radar and the infrared camera, the detection range (position) of the two shifts, and a pedestrian in the infrared image corresponding to the detection information of the laser radar is detected. There are cases where it cannot be done. This synchronization shift occurs because the detection time (for example, about 120 ms) of the object by the laser radar accompanied by scanning becomes longer than the acquisition time (for example, about 33 ms) of the image including the object by the infrared camera.

  In the apparatus described in Patent Document 1, no consideration is given to the correction for the synchronization deviation between the laser radar and the infrared camera.

  Therefore, the present invention improves the extraction accuracy of a predetermined object on the camera image specified by the radar by detecting and correcting a synchronization shift (phase shift) between the output signals of the radar and the camera from the obtained image information. The purpose is to let you.

  The present invention provides a radar that detects an object around a vehicle, a camera that captures an image around the vehicle, and a predetermined area on a corresponding image captured by the camera that corresponds to the position of the object detected by the radar. There is provided a vehicle periphery monitoring device including means for specifying and specifying the object in the predetermined area. The periphery monitoring device calculates a phase shift amount between a temporal change in the movement amount of the predetermined area on the identified corresponding image and a temporal change in the movement amount of the predetermined object on the image captured by the camera; Phase correction means for correcting the phase of the output signal of the radar or camera so that the phase shift amount becomes zero.

  According to the present invention, on the camera image specified by the radar caused by the synchronization shift (phase shift) between the radar and the output signal of the camera by monitoring the time change of the movement amount on the image captured by the camera. It is possible to correct the positional deviation between the position of the area and the position of the predetermined object imaged by the camera, and as a result, it is possible to improve the specification (extraction) accuracy of the predetermined object on the camera image specified by the radar. It becomes possible.

  According to one aspect of the present invention, the phase correction means performs phase correction when the amount of time change in the relative position between the vehicle and the predetermined object is greater than or equal to the predetermined amount.

  According to one aspect of the present invention, the time change in the relative position between the vehicle and the predetermined object is monitored by monitoring the time change in the movement amount on the image, thereby synchronizing the output signals of the radar and the camera. Deviation (phase deviation) correction can be reliably performed at a more stable and appropriate timing.

  According to an aspect of the present invention, the phase correction unit combines time changes in the positions of the plurality of objects that occur as the vehicle travels when the radar detects a plurality of objects while the vehicle is traveling. The phase correction is performed when the amount of change obtained in this way is greater than or equal to a predetermined amount.

  According to one aspect of the present invention, a section (time) in which synchronization deviation correction is performed is performed after capturing the movements (movement amounts) of a plurality of stationary objects detected by a radar as the vehicle travels. This makes it possible to reliably perform synchronization deviation (phase deviation) correction between the output signals of the radar and the camera at a more stable and appropriate timing.

  According to one aspect of the present invention, the phase correction unit obtains the time change amount of the relative position by combining the sections where the time change of the relative position is equal to or longer than the predetermined time.

  According to one aspect of the present invention, it is possible to obtain necessary and sufficient data capable of detecting the synchronization deviation between the output signals of the radar and the camera by combining the time variation amounts of a plurality of relative positions, and improving the detection accuracy. It becomes possible.

It is a block diagram which shows the structure of the periphery monitoring apparatus of the vehicle of this invention. It is a figure which shows the processing flow in the image processing unit of this invention. It is a figure which shows the acquisition flow of the time series data of the variation | change_quantity of this invention. It is a figure which shows the predetermined area | region and pedestrian image on the image according to one Example of this invention. It is a figure which shows the calculation flow of the phase shift amount of this invention. It is a figure which shows the flow of the phase correction of this invention. It is a figure for demonstrating acquisition of the time series data of the variation | change_quantity of this invention. It is a figure for demonstrating acquisition of the time series data of the variation | change_quantity of this invention. It is a figure for demonstrating the calculation method of the phase shift amount of this invention.

  Embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a vehicle periphery monitoring device according to an embodiment of the present invention. The periphery monitoring device is mounted on a vehicle, and includes a laser radar 10, an infrared camera 12, an image processing unit 14 for detecting an object around the vehicle based on image data captured by the infrared camera 12, and a detection result thereof. And a display device 18 for displaying an image obtained through imaging by the infrared camera 12 and for causing the driver to recognize an object around the vehicle. With.

  The laser radar 10 corresponds to, for example, a scanning laser radar, and receives the reflected light from the object of the beam scanned in each direction, thereby detecting the position (region) of the object (candidate) in each direction as a detection point. Detect the distance to it. The laser radar 10 may be another type of radar (for example, a millimeter wave radar). The infrared camera 12 may also be a camera that uses light in a wavelength band other than infrared (for example, visible light). Therefore, in the following description, a gray scale image taken by the infrared camera 12 is targeted, but the present invention is not limited to this. Furthermore, in a vehicle provided with a navigation device, the corresponding function provided in the navigation device may be used as the speaker 16 and the display device 18.

  The image processing unit 14 in FIG. 1 includes (executes) means (functions) indicated by blocks 141 to 145. That is, the image processing unit 14 receives a signal from the laser radar 10 and specifies an object position specifying unit 141 for specifying the position of an object around the vehicle, and an image around the vehicle captured by the infrared camera 12 as a grayscale image. An image acquisition unit 142 to acquire, and a unit 143 that specifies a predetermined region on the acquired corresponding grayscale image corresponding to the position of the object specified by the object position specifying unit 141 and specifies an object in the predetermined region. including. The object position specifying unit 141 may be configured as a part of the function of the laser radar 10, that is, integrated with the laser radar 10.

  The image processing unit 14 further includes a phase change between the temporal change in the movement amount of the predetermined area on the corresponding image specified by the laser radar 10 and the temporal change in the movement amount of the predetermined object on the image captured by the infrared camera 12. Means 144 for calculating the amount of deviation and phase correction means 145 for correcting the phase of the output signal of the radar or camera so that the amount of phase deviation becomes zero.

  The image processing unit 14 also needs to receive detection signals from a vehicle speed sensor that detects the speed (vehicle speed) of the host vehicle, an acceleration sensor, and a yaw rate sensor that detects a yaw rate (change speed of the rotation angle in the turning direction). Has a function to perform various processing.

  The function of each block is realized by a computer (CPU) included in the image processing unit 14. The configuration of the image processing unit 14 may be incorporated in the navigation device.

  The image processing unit 14 includes, for example, an A / D conversion circuit that converts an input analog signal into a digital signal, an image memory that stores a digitized image signal, and a central processing unit (CPU) that performs various arithmetic processes. ), RAM used by the CPU to store data for computation, ROM to store programs executed by the CPU and data to be used (including tables and maps), drive signals for the speakers 16, display signals to the display device 18, etc. An output circuit for outputting is provided. The output signals of the laser radar 10 and the infrared camera 12 are converted into digital signals and input to the CPU.

  Next, a method for correcting the phase of the output signal of the radar or camera in one embodiment of the present invention will be described.

  FIG. 2 is a diagram illustrating a processing flow executed by the image processing unit 14. The processing flow in FIG. 2 is executed at predetermined time intervals by calling the processing program stored in the memory by the CPU of the image processing unit 14. In step S1 of FIG. 2, time-series data is acquired in order to specify a data section for performing synchronization (phase) shift correction.

  FIG. 3 is a diagram showing a flow (routine) for acquiring time-series data of the change amount in step S1 of FIG. In step S11 of FIG. 3, the position of a predetermined object around the vehicle is specified by receiving a signal from the laser radar 10. An arbitrary method can be used for the identification, for example, as follows.

  A scanning laser radar is used as the laser radar 10, and the position of an object, for example, a pedestrian candidate is specified from the width of the detection point group. Specifically, the scanning laser radar measures the distance to the object in each direction as a detection point by receiving reflected light from the object of the beam scanned in each direction. A set of detection points is obtained by plotting the positions of these detection points in a two-dimensional coordinate system in which the position of the laser radar is the origin and the Y axis is the front direction of the radar. From these set of detection points, detection point groups are grouped as detection point groups whose mutual distance between detection points is equal to or smaller than a predetermined value, and among the grouped detection point groups, those having a spread width equal to or smaller than a predetermined value are grouped as predetermined objects. The position is identified as a candidate (for example, a pedestrian candidate).

  In step S12, an output signal (that is, captured image data) of the infrared camera 12 is received as input, A / D converted, and stored in the image memory. The stored image data is a grayscale (monochrome shade) image including luminance information. In an infrared camera in which the higher the temperature of the target, the higher the brightness is, the image data has a higher brightness value in the higher brightness portion, and the image data has a lower brightness value as the temperature becomes lower (lower brightness).

  In step S13, a predetermined area that is a range in which a predetermined object is detected is set in the acquired grayscale image. This predetermined area is an area including the position (area) of the predetermined object candidate specified by the laser radar 10 in step S12. An arbitrary method can be used for setting the predetermined area. For example, when the position of the pedestrian is specified as the position of the object, an area of a predetermined size including the position of the pedestrian candidate (detection point) is used. Group) is converted into the camera coordinate system of the infrared camera 12, mapped (superimposed) in the gray scale image, and set as a predetermined area on the corresponding gray scale image.

  In step S14, a predetermined object on the image is specified. This specification is performed in the area or in the vicinity of the predetermined area on the image set in step S13. This specification can be performed by an arbitrary method. For example, an object is collated from a black and white image obtained by binarizing a grayscale image. Specifically, first, it is checked whether or not it has the characteristics of the excluded object stored in the memory in advance. The excluded object means, for example, an object (a quadruped animal, a vehicle, etc.) that is a heating element but has a clearly different shape or the like. If it does not correspond to the excluded object, it is checked whether it corresponds to the characteristic of the predetermined object. For example, when the predetermined object is a pedestrian, a rounded white portion exists as a feature of the pedestrian's head, and an elongated white portion corresponding to the body and leg portions is connected below the white portion. Etc. are included. Further, the similarity with a predetermined pattern representing a pedestrian stored in advance may be calculated using known pattern matching, and the similarity may be used as a material for determining whether or not the user is a pedestrian. Furthermore, since the walking speed of the pedestrian is substantially constant, information on the moving speed of the object obtained from the laser radar 10 may be used as one of judgment materials.

  In step S15, it is determined whether a predetermined object has been identified. If this determination is No, the process ends and returns. When the determination in step S15 is Yes, the process proceeds to step S16. In step S16, the center of gravity of the predetermined area set in step S13 is calculated. In step S17, the center of gravity of the pedestrian image specified in step S14 is calculated.

  The calculation of the center of gravity in steps S16 and S17 will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a predetermined area and a predetermined object (pedestrian) on a black and white image after binarization processing. In FIG. 4, in order to recognize the line, the whole is white except for the line part, but the background part around the pedestrian images 22 and 24 that are the predetermined objects is actually displayed in black.

  In FIG. 4, pedestrian images 22 and 24 are shown as predetermined objects in the binarized black and white image 20 acquired by the infrared camera 12. The image 22 of (a) is a case where the pedestrian is stationary, and the image 24 of (b) is equivalent to the case where the pedestrian is moving in the right direction in the figure. Frames indicated by reference numerals 21 and 23 are predetermined areas specified by the laser radar 10 as pedestrian positions (areas). While the pedestrian image 22 in (a) is within the predetermined area 21 and the positional deviation between the two is small, the positional deviation between both is large in (b). The positional deviation in (b) is mainly caused by a delay in the output signal (processing) of the laser radar 10, and corresponds to a case where the synchronization deviation between the radar and the camera is relatively large. On the other hand, the case (a) corresponds to a case where there is little or little synchronization deviation between the two. The case where the synchronization error correction is particularly necessary is the case as shown in (b).

  4A, the center of gravity of the predetermined area 21 is calculated on the position Q1, and the center of gravity of the pedestrian image 22 is calculated on the position P1. The center of gravity of both is displaced by the amount of positional deviation d1, but the amount of positional deviation is small. Similarly, in (b), the center of gravity of the predetermined area 23 is calculated on the position Q2, and the center of gravity of the pedestrian image 24 is calculated on the position P2. The center of gravity of both is shifted by a positional shift amount d2, and the positional shift amount is large. Each center of gravity is calculated at each predetermined timing, and the position information and the time change amount are stored as data in a memory. Further, in steps S16 and S17, a position determined by another arbitrary reference instead of the center of gravity, for example, the midpoint of the predetermined area and the maximum horizontal width of the pedestrian image may be calculated.

  In step S18 of FIG. 3, it is determined whether or not the time change amounts of the center of gravity of the predetermined region and the center of gravity of the object are larger than the predetermined amount. In the example of FIG. 4, it is determined whether or not the amount of time change of each of the center of gravity of the predetermined area 23 and the center of gravity of the pedestrian image 24 is larger than the predetermined amount in (b) where correction of synchronization deviation is particularly necessary. . This determination is performed in order to use only data at a relatively large amount of time change for phase shift detection. If this determination is No, the process ends and returns. If this determination is Yes, in step S19, the data of the center of gravity of the predetermined area (including the time change amount) is stored in the memory as time series data. Similarly, in subsequent step S20, the center-of-gravity data (including the time change amount) of the object is stored in the memory as time series data.

  Returning to FIG. 2, in step S <b> 2, it is determined whether the amount of time-series data of the predetermined region and the center of gravity obtained in step S <b> 1 is equal to or greater than a predetermined amount. This determination is performed in order to obtain a state in which time variation data is sufficiently accumulated and phase shift detection can be effectively obtained. If this determination is No, the process returns to step S1, and data acquisition is repeated until time series data of a predetermined amount or more is obtained.

  Here, the time series data acquisition of the predetermined amount or more will be further described. The phase shift is corrected by detecting the phase shift between the radar and the camera during the period when the phase shift between the radar and the camera is relatively large, that is, the amount of temporal position change of the object detected by the radar or camera is relatively large. It is preferable from the viewpoint of detection accuracy and the like to detect the amount of phase shift in an easy period. However, for example, when the vehicle is traveling on a general road, it is difficult to continuously and stably observe changes necessary for the correction. Therefore, for example, the following method is used to acquire time-series data in which data changes over time over a certain period of time.

  FIG. 7 is a diagram for explaining that the continuous sections A ′ to D ′ are created by gathering only the sections of the sections A to D where the temporal position change amount of the object detected by the radar or the camera is large. . In FIG. 7, the sections A to D having a large temporal position change amount shown on the upper side are gathered together to create continuous sections A ′ to D ′ at the time T1 on the lower side of the figure. The section where the amount of change in position of the object in FIG. 7 is large is various detection signals indicating the running state of the vehicle when the change in the vehicle state is large, such as the yaw rate during turning of the vehicle or the vehicle speed during acceleration / deceleration. It is desirable to obtain from In this way, by combining only sections where the amount of change in the position of the object over time is large, and detecting and correcting the phase shift between the laser and camera in the combined section, the accuracy and reliability of the correction Can be improved.

  FIG. 8 is a diagram for explaining another example of acquiring time-series data of a predetermined amount or more. In an in-vehicle environment in which the host vehicle moves (runs), stationary targets are frequently replaced, so it is difficult to continuously and stably observe changes necessary for correction on a general road. Therefore, in the example of FIG. 8, temporally continuous data is generated by connecting the moving speeds of different stationary targets (pedestrians and the like).

  In FIG. 8A, there are three stationary targets 1 to 3 in the traveling direction of the vehicle 30. As the vehicle 30 travels, the three stationary targets 1 to 3 gradually approach the vehicle 30. (B) shows the speed change 32 of the vehicle 30 and the speed change 33 of the stationary targets 1 to 3 in that case. As illustrated, the speed change 33 of the stationary targets 1 to 3 is obtained by connecting the speed changes A to C of the stationary targets 1 to 3, and is symmetric (negative) with respect to the speed change 32 of the vehicle 30. Relationship (graph). Thereby, it is possible to obtain a section in which the relative speed of the vehicle and the stationary target changes. Similarly to the case of FIG. 7, by detecting and correcting the phase shift amount of the laser and the camera in the section, The accuracy and reliability of the correction can be improved.

  After the determination in step S2 of FIG. 2 is Yes, in step S3, a routine for calculating the phase shift amount between the output signals of the radar and the camera is executed. FIG. 5 is a diagram illustrating a calculation flow of the phase shift amount. In step S31, a phase shift amount ΔT is calculated from time series data of the center of gravity of the predetermined area and the center of gravity of the object.

  FIG. 9 is a diagram for explaining a method of calculating the phase shift amount ΔT. (A) is a binarized black and white image 20 obtained by extracting the part (b) of FIG. 4, and the same reference numerals as those in FIG. 4 are given. As already described in the description of FIG. 4, the predetermined region 23 specified as the position (region) of the pedestrian by the laser radar 10 is more than the pedestrian image 24 after the binarization processing acquired by the infrared camera 12. Displayed out of position.

  FIG. 9B is a diagram showing a time change A of the movement amount of the pedestrian image 24 on the monochrome image 20 and a time change B of the movement amount of the predetermined area 23 corresponding to the case of FIG. The data (b) is a part of data obtained by cutting out each movement amount data in a continuous section having a relatively large change as obtained by the example of FIG. 7 or FIG. From the graph of (b), it can be seen that the movement amount change B of the predetermined region 23 is delayed in phase (signal rising) by ΔT1 time than the movement amount change A of the pedestrian image 24. Therefore, the phase shift amount in this case is ΔT1. This phase shift amount ΔT1 corresponds to the positional shift amount d2 on the image of (a).

  As described above, by monitoring the temporal change in the movement amount of the predetermined area on the image and the predetermined object image in the predetermined section (time), the phase shift amount of both can be obtained. This amount of phase shift corresponds to the amount of synchronization shift between the output signals of the laser radar 10 and the infrared camera 12. Instead of looking at the difference between the rising times of the two movement amount changes, the phase deviation amount may be obtained from the period of the obtained function by obtaining the cross-correlation coefficient between them or using the phase-only correlation method. .

  In step S4 of FIG. 2, a routine for performing phase correction is executed. FIG. 6 is a diagram showing a flow of the phase correction. In step S41, it is determined whether or not the phase shift amount ΔT obtained in step S41 is larger than a predetermined threshold value ΔTth. The threshold value ΔTth is set so that it can be selected only when there is a phase shift amount that requires phase correction. For example, in the example of FIGS. 4 and 9 described above, the phase shift amount ΔT1 in FIG. 4B corresponds to the case where the phase shift amount ΔTth is larger than the threshold value ΔTth, and the phase shift amount in FIG. This is the case when it is smaller. If this determination is No, the process ends and returns. If the determination is Yes, the process proceeds to the next step S42.

  In step S42, the object position on the camera image or the specific area (predetermined area) by the radar is shifted by ΔT. Specifically, in the processing of the output signal of the laser radar 10 and the image processing of the output signal of the infrared camera 12, correction (adjustment) is performed so that the phase shift amount ΔT1 in FIG. 9B is eliminated (becomes zero). For example, the rise time of the output signal of the infrared camera 12 is delayed by an amount corresponding to the phase shift amount ΔT1, or the output signals of the laser radar 10 and the external camera 12 rise in synchronization with a predetermined clock (timing) signal. Perform processing such as.

  The embodiment of the present invention has been described above, but the present invention is not limited to such an embodiment, and can be modified and used without departing from the spirit of the present invention.

10 Laser radar,
12 Infrared camera,
14 Image processing unit 16 Speaker,
18 display device,
20 Camera images 21 and 23 Predetermined areas 22 and 24 detected by the radar Pedestrian image 30 Vehicle

Claims (4)

A radar that detects an object around the vehicle, a camera that captures an image around the vehicle, and a predetermined area on the corresponding image captured by the camera corresponding to the position of the object detected by the radar A vehicle surroundings monitoring device comprising means for identifying the object in a predetermined area,
A first time change waveform of a change amount per predetermined time of a position of a predetermined region on the corresponding image repeatedly specified according to a time change of a relative position of the object with respect to the vehicle, and the camera repeatedly captured means for calculating the position of the front SL product body in the image, the second time change waveform of change per predetermined time, the phase shift amount is a displacement amount in the time axis direction,
The output signal of the previous SL radar or the camera, shifted by one another wherein the phase shift amount is represented time, a phase correction means for the phase shift amount to perform correction to be zero,
The a, the vehicle surroundings monitoring apparatus. The phase correction means corrects the phase when the amount of change of the position of the predetermined region on the image per predetermined time or the amount of change of the position of the object on the image per predetermined time is greater than or equal to a predetermined amount. The periphery monitoring apparatus according to claim 1, wherein Said phase correction means, during traveling of the vehicle, when the radar detects a plurality of objects, said plurality of each object, per unit time of a relative position with respect to the vehicle which occurs with the progress of the vehicle Combining a time-varying waveform of a change amount of each of the two, specifying a period in which the change amount changes with time, and performing the phase correction based on output signals of the radar and the camera in the specified period. Item 1. The periphery monitoring device according to Item 1. It said phase correction means, for either the first or second time varying waveforms, the change amount of the position of the position or the object of the predetermined area per predetermined time is equal to or more than a predetermined time period the amplitude of the time change waveform A period that is continuously greater than or equal to a predetermined amount is specified, and the first and second time change waveforms in the specified periods are combined with each other , and the first and second time change waveforms after the combination are combined. The periphery monitoring apparatus according to claim 2 , wherein the phase correction is performed based on the information.

Images