JP5642049B2 - Vehicle external recognition device and vehicle system using the same - Google Patents

Description

  The present invention relates to a vehicle external environment recognition device and a vehicle control system using the same, for example, a vehicle external environment recognition device that detects an object around the vehicle based on image information from an in-vehicle camera (imaging device) and the like. The present invention relates to a vehicle system using the.

  In order to reduce the number of casualties due to traffic accidents, the development of preventive safety systems that prevent accidents in advance is underway. A preventive safety system is a system that operates in a situation where there is a high possibility of an accident.For example, when there is a possibility of collision with an obstacle in front of the host vehicle, a warning is given to the driver to alert the driver. Pre-crash safety systems have been put into practical use that reduce the damage to passengers by automatic braking when inevitable situations occur.

  As a method of detecting obstacles around the host vehicle, a method of processing and detecting time-series images captured by a camera is known. For example, Patent Document 1 describes a method of detecting an object from movement between two images captured by an in-vehicle camera.

  When detecting an object using time-series motion in an in-vehicle camera, a method of extracting a region that moves differently from the motion estimated from the own vehicle speed is assumed, assuming a distance to the background stationary object or the road surface. The position of the background stationary object and the road surface on the image changes according to the vehicle speed, and the change is larger as the vehicle speed is higher, and the change is smaller as the vehicle speed is lower. Therefore, when using images captured at a certain period, the motion on the background stationary object or road surface image increases as the vehicle speed increases, so the processing load is increased and the vehicle speed is low. In this case, since the motion between images is small, it is difficult to detect an object. In order to solve this, it is conceivable to change the timing for capturing an image according to the speed of the vehicle.

  Regarding the technology for changing the imaging timing according to the own vehicle speed, in Patent Document 2, the acquisition timing is changed according to the vehicle speed in the camera dirt detection process that is activated at regular intervals for a monocular camera attached to the vehicle. A technique for detecting dirt attached to a camera from a difference between captured images is described.

JP 2008-3695 A JP 2003-259358 A

  However, it is important to execute at a fixed processing cycle in detecting an obstacle or the like related to vehicle warning or control. In the combination of the above methods, when processing is performed at a constant cycle, the processing is not synchronized with the image capture cycle, and therefore processing is performed using an image that is not captured at the latest timing, resulting in an object detection delay.

  In addition, there is a method of capturing an image with a very short cycle and saving it in a memory, and selecting an image at the time of processing. However, since there is a limit to the capture cycle, an image with an optimal timing is always captured. However, it is not preferable because a large amount of memory is used.

  The present invention has been made in view of such problems, and an object of the present invention is to provide an external environment recognition device for a vehicle that captures an optimal image when observing the movement of an object at a constant processing cycle, and It is to provide a vehicle system using the same.

In order to achieve the above object, an external environment recognition device for a vehicle according to the present invention includes an image capturing unit that captures an image from an imaging device such as a camera, an own vehicle behavior acquisition unit that acquires an own vehicle speed, and an image from the imaging device. An image capturing timing adjusting unit that adjusts an image capturing timing, and the image capturing unit captures an image at a fixed period set by the image capturing timing adjusting unit, and captures an image at irregular intervals during the fixed period. The imaging timing adjustment means adjusts the timing of irregular image capture according to the vehicle speed, separates the foreground and background using the captured image, and detects a collision from the foreground. In a device that detects possible objects, adjust the timing for capturing images according to the behavior of the vehicle, and capture images at a certain period. , It is characterized in that there is an image capture of irregularly during the period.

  According to the present invention, an image captured at a timing of a fixed processing cycle and an image captured at an imaging timing adjusted according to the vehicle speed irregularly during the cycle are captured and motion is detected. Therefore, the object information output at a fixed processing cycle can be detected from the latest image motion information.

The block diagram of the external field recognition apparatus for vehicles in 1st Embodiment of this invention. The schematic diagram which shows the example of a process of the imaging timing adjustment part in 1st Embodiment of this invention. The flowchart showing the process of the foreground / background separation unit in the first embodiment of the present invention. The flowchart showing the process of the object detection part in 1st Embodiment of this invention. The schematic diagram which shows the example of the process of the object detection part in 1st Embodiment of this invention. The flowchart showing the process of the object detection part in 1st Embodiment of this invention. The block diagram of the external field recognition apparatus for vehicles in 2nd Embodiment of this invention. The block diagram of the external field recognition apparatus for vehicles in 3rd Embodiment of this invention. The flowchart showing the process of the estimated image generation part in 3rd Embodiment of this invention. The flowchart showing the process of the foreground / background separation unit in the third embodiment of the present invention. The block diagram of the external field recognition apparatus for vehicles in 4th Embodiment of this invention. The schematic diagram which shows the example of a process of the imaging timing adjustment part in 4th Embodiment of this invention. The flowchart showing the process of the estimated image generation part in 4th Embodiment of this invention. The flowchart showing the process of the vehicle system using the external field recognition apparatus for vehicles of this invention. The schematic diagram which shows the example of calculation of the risk degree of the vehicle system using the external field recognition apparatus for vehicles of this invention.

<First embodiment>
Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram of a vehicle external environment recognition apparatus 1000 according to the first embodiment. This embodiment corresponds to claims 1, 2, 4, 5, 6, 8, and 9.

  The vehicle external environment recognition device 1000 is incorporated in a camera device mounted on an automobile, an integrated controller, or the like, and detects an object from an image captured by a camera 1010 of the camera device. In the form, it is configured to detect obstacles, vehicles and pedestrians in front of the host vehicle.

  The vehicle external environment recognition apparatus 1000 is configured by a computer having a CPU, a memory, an I / O, and the like, programmed with predetermined processes and procedures, and repeatedly executes processes at a predetermined period T.

  As shown in FIG. 1, the vehicle external environment recognition apparatus 1000 includes an image acquisition unit 1011, an own vehicle behavior acquisition unit 1021, an imaging timing adjustment unit 1031, an optical flow calculation unit 1041, a foreground / background separation unit 1051, An object detection unit 1061.

  The image acquisition unit 1011 captures image data obtained by capturing the front of the host vehicle from a camera 1010 attached to a position where the front of the host vehicle can be imaged, and acquires a fixed period acquired image IMGSRC [x] [y] and an irregularly acquired image. IMGSRC_SUB [x] [y] is stored in the RAM 1020. In addition, the fixed period acquisition image IMGSRC [x] [y] and the irregular acquisition image IMGSRC_SUB [x] [y] are two-dimensional arrays, and x and y indicate the coordinates of the image data, respectively.

  The fixed period acquisition image IMGSRC [x] [y] is captured at a predetermined fixed period acquisition timing. In the present embodiment, it is assumed that the data is captured at the same timing as the processing cycle T. Also, the irregularly acquired image IMGSRC_SUB [x] [y] is acquired during the processing cycle T, and the acquisition timing is determined by the imaging timing adjustment unit 1031. The RAM 1020 stores the periodic acquisition image IMGSRC [x] [y] and the irregular acquisition image IMGSRC_SUB [x] [y] acquired by the image acquisition unit 1011.

  The own vehicle behavior acquisition unit 1021 acquires a detection signal of a vehicle speed measurement device mounted on the own vehicle, and acquires the own vehicle speed VSP. The own vehicle speed VSP may be acquired by directly inputting a signal of the vehicle speed measuring device to the vehicle external environment recognition device 1000, or may be acquired by performing communication using a LAN (Local Area Network).

  The imaging timing adjustment unit 1031 uses the own vehicle speed VSP acquired by the own vehicle behavior acquisition unit 1021 to adjust the imaging timing of the image acquisition unit. Specifically, it has a function of setting an imaging timing for capturing a fixed-period acquired image and an irregularly acquired image. Details of the processing will be described later.

  The optical flow calculation unit 1041 acquires the fixed period acquisition image IMGSRC [x] [y] and the indefinite period acquisition image IMGSRC_SUB [x] [y] from the RAM 1020, and calculates the optical flow between the images. The optical flow is a vector indicating to which point in one image the coordinate (x, y) has moved to one image, and is in the X direction with respect to one image coordinate (x, y). The component and the component in the Y direction are calculated. The calculated optical flow is stored on the RAM 1020. There are two methods for storing such as an optical flow vector expressed as an image and an optical flow measured at N points as an array OFX [n] and OFY [n]. In the embodiment, the data are stored as arrays OFX [n] and OFY [n]. Details of the processing will be described later.

  The foreground / background separator 1051 analyzes the optical flows IMGOFX [x] [y] and IMGOFY [x] [y], and separates them into the foreground and the background. In the present embodiment, the foreground areas on the image are grouped and expressed as FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g]. Here, g is an ID number when a plurality of foregrounds are extracted. Details of the processing will be described later.

  The object detection unit 1061 specifies an object from the foreground areas FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g], and calculates a distance and a horizontal position. In the present embodiment, a case where a vehicle is detected from the foreground and a case where a pedestrian is detected will be described. Details of the processing will be described later.

[Imaging timing adjustment unit]
Next, the contents of processing in the imaging timing adjustment unit 1031 will be described with reference to FIG. FIG. 2 is a timing chart for explaining the imaging timing of the image acquisition unit 1011.

  The imaging timing adjustment unit 1031 determines the next imaging timing T1 and the next imaging timing T2. Here, the imaging timing T2 is determined so that an object on the image captured at the imaging timing T1 moves by a predetermined amount when it appears on the image captured at a fixed period T. For example, when T2 is determined so that all stationary objects move by P meters, it can be determined from T2 = P / VSP. T1 is obtained from T1 = T−T2 using the processing cycle T.

  Further, there is a method of calculating the imaging timing from the maximum detection distance of the camera. If the maximum sensing distance at which an object is to be detected is YMAX, the movement will be smaller on the image, so the farther away the object moves from the YMAX point, the greater the distance from which the object will move, the sufficient movement on the image will be investigated in advance. Therefore, the imaging timing T2 can be determined from the distance and the own vehicle speed VSP.

  Note that thresholds T2Min and T2Max are provided in order to cope with the case where VSP is low and T2> T, or when VSP is high and T2 exceeds the limit of the capturing speed of the image sensor, and when T2> T2Max, T2 = In the case of T2Max and T2 <T2Min, a process for setting T2 = T2Min is necessary.

[Optical flow calculation unit]
Next, the contents of processing in the optical flow calculation unit 1041 will be described.
The optical flow calculation unit 1041 sets N points (x0, y0) to (xN−1, yN−1) on the latest captured image IMGSRC [x] [y], and each point is one point before the time point. An optical flow representing the position on the image IMGSRC_sub [x] [y] captured at the timing is calculated. In this embodiment, the N point is a fixed point that is arranged in advance in a grid pattern with an interval of width Wp and height Hp. For example, a point calculated by a feature point extraction process described in a known document is used for each frame. It may be used. Of course, you may calculate by all the coordinate points on an image.

  The calculation of the optical flow is variously introduced in known literatures, but in this embodiment, the block match method is used. Since the block match method is described in known documents relating to image processing, a detailed description thereof is omitted, but a template image TP of a small region centered on a point (xn, yn) on the image IMGSRC [x] [y]. , Search for a position (xn ′, yn ′) that best matches the template image TP in the image IMGSRC_sub [x] [y], and x-direction component OFX [n] = (x1′−x1), This is a method of calculating the y-direction component OFY [n] = (yn′−yn).

  As described above, in motion vector detection that operates at a constant processing cycle, since a motion vector can be detected from an image captured at the timing of the processing cycle, there is no information delay and the processing cycle is constant. Since the acquisition timing of the image captured at this timing and the image captured irregularly before that is adjusted according to the vehicle speed, the background motion vector in a certain scene can be detected constantly regardless of the vehicle speed. .

[Foreground / Background Separation Unit 1051]
Next, the contents of processing in the foreground / background separation unit 1051 will be described with reference to FIG. FIG. 3 is a flowchart showing a processing flow of the foreground / background separation unit 1051. First, in step S301, a loop control variable n = 0 is set.

  Next, in step S302, the position of the point (xn, yn) on the latest captured image IMGSRC [x] [y] is predicted on the image T2 seconds before, and the prediction flow is calculated. To do. In the present embodiment, the prediction flow is obtained by the following procedure. First, from the vanishing point and the camera geometry, assuming that the coordinates (xn, yn) are points on the road surface, the world coordinates (Xn, Yn, 0) are obtained, and the movement amount P = VSP from the own vehicle speed VSP and the capture timing T2. XT2 is calculated to obtain the predicted position (Xn, Yn + P, 0), and the image coordinate position (xnp, ynp) of the predicted position (Xn, Yn + P, 0) is calculated. The prediction flow is PFX = (xnp−xn), PFY = (ynp−yn).

Next, in step S303, an error e between the prediction flows PFX, PFY and OFX [n], OFY [n] is obtained. In the present embodiment, the error e is the Euclidean distance between the flow vectors, and the error e is obtained by the following equation.
e = (OFX [n] −PFX) ^ 2 + (OFY [n] −PFY) ^ 2

  Next, in step S304, the error e is compared with the threshold value th. If the error e is larger than the threshold value, the process proceeds to step S305. (Xn, yn) is determined as the foreground, and if it is equal to or less than the threshold value, the process proceeds to step S306 (xn). , Yn) is determined as the background.

  Next, step S302 and subsequent steps are repeated until the loop control variable n reaches N in step S307. With the above processing, it is determined whether the foreground or the background from the point (x0, y0) to the point (xN-1, yN-1).

Next, in step S308, the points determined as the foreground are grouped to obtain the foreground regions FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g]. The grouping is performed as follows, for example. In the present embodiment, since the points where the optical flow is calculated at intervals of the width W and the height H are arranged in a grid pattern, first, the points determined as the foreground are (xn / W, yn / H). Convert and map on 2D map. Then, adjacent points on the two-dimensional map are grouped. If the upper, lower, left, and right ends of each group on the two-dimensional map are GR_SX [g], GR_SY [g], GR_EX [g], and GR_EY [g], the foreground area is obtained by the following equation.
FG_SX [g] = Wp × GR_SX [g] −Wp / 2
FG_SY [g] = Hp × GR_SY [g] −Hp / 2
FG_EX [g] = Wp × GR_EX [g] −Wp / 2
FG_EY [g] = Hp × GR_EY [g] −Hp / 2

  Through the above processing, the foreground areas FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g] are obtained.

[Object detection unit 1061]
Next, the contents of processing in the object detection unit 1061 will be described. In the present embodiment, as the object detection unit 1061, two patterns for detecting a vehicle and detecting a pedestrian will be described.

  First, processing for detecting a vehicle by the object detection unit 1061 will be described with reference to FIG. FIG. 4 is a flowchart showing a process flow of the object detection unit 1061. First, in step S401, the control variable g = 0 is set.

  Next, in step S402, a vehicle is detected in the foreground areas FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g]. As described above, the foreground regions FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g] are obtained by the optical flow arranged at intervals of the width W and the height H. It is expected that information other than vehicles is also included. Therefore, as in the method described in Japanese Patent Application Laid-Open No. 2005-156199, a method of determining the position of the vehicle by analyzing the edge, or a method of detecting the vehicle by repeating pattern matching while changing the position / size within the region is used. .

  Since a known technique may be used for pattern matching, a detailed description is omitted here. For example, there is a method using a neural network as follows. In this method, as shown in FIG. 5, first, an image is extracted from an image region to be determined whether or not it is a vehicle, and reduced to a predetermined size to generate a reduced image 501. In this embodiment, the image is reduced to an image having a width of 16 pixels and a height of 12 pixels. The pixels of the reduced image are raster scanned to generate a one-dimensional vector 502, which is used as an input to the neural network 503.

  The neural network 503 imitates a human brain network, and includes an input layer 5031, an intermediate layer 5033, and an output layer 5035 composed of a plurality of nodes, and each node of the input layer 5031 and the intermediate layer 5033. A weighting factor 5032 exists between the nodes, and a weighting factor 5034 exists between each node of the intermediate layer 5033 and each node of the output layer 5035. The output of the neural network is one value of the node of the output layer, and this value is obtained by the product-sum operation of the values of the nodes of all the intermediate layers 5033 connected to this node and their weight coefficients. Furthermore, the value of each node in the intermediate layer 5033 is obtained by multiply-adding the values of the nodes of all input layers connected to each node and their weighting coefficients.

  Now, in the vehicle pattern detection, since the one-dimensional vector 502 is directly connected to the input layer, the value of each node in the output layer can be calculated by the above-described processing. As a result, if the value of the predetermined node of the output layer exceeds the threshold value, it is determined that the vehicle pattern exists.

  Predetermined nodes in the output layer must be determined when creating a program in advance. When a vehicle pattern enters the input node, the output of the node in the output layer exceeds the threshold value. When the above pattern is entered, the weighting coefficient between the nodes needs to be adjusted in advance so that the output is below the threshold value. As a method of adjustment, a back propagation method that is a known technique may be used.

  Next, if there is an area determined to be a vehicle in step S403, the process moves to step S404 to calculate a physical parameter. On the other hand, if the determined area does not exist, the process proceeds to step S405.

  In step S404, the physical parameter of the detected vehicle is calculated from the detected position on the vehicle image. Specifically, if the detected position of the vehicle on the image is V_SX [i], V_SY [i], V_EX [i], V_EY [i], the distance PYO is calculated using the lower end of the area and the camera geometry. [i] is calculated, and the lateral position PXO [i] is calculated from the center position CX = (V_SX [i] + V_EX [i]) / 2. Here, i is an ID number when a plurality of vehicles are extracted.

  Next, step S402 and subsequent steps are repeated until the loop control variable g reaches G in step S405. The vehicle can be detected by the above processing.

  Next, processing for detecting a pedestrian by the object detection unit 1061 will be described with reference to FIG. FIG. 6 is a flowchart showing a process flow of the object detection unit 1061. First, in step S601, the control variable g = 0 is set.

  In step S602, a pedestrian is detected from the foreground areas FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g]. As described above, the foreground regions FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g] are obtained by the optical flow arranged at intervals of the width W and the height H. Information other than pedestrians is also expected to be included. Therefore, a method of detecting a pedestrian by repeating pattern matching while changing the position and size in the region is used. In addition, since a well-known technique may be used about pattern matching, detailed description is omitted here.

  Next, if there is an area determined to be a pedestrian in step S603, the process moves to step S604 to calculate physical parameters. On the other hand, if the determined area does not exist, the process proceeds to step S605.

  In step S604, the physical parameter of the detected pedestrian is calculated from the detected position on the pedestrian image. Specifically, if the detected position of the pedestrian on the image is P_SX [i], P_SY [i], P_EX [i], P_EY [i], the distance using the lower end of the region and the camera geometry PYO [i] is calculated, and the lateral position PXO [i] is calculated from the center position CX = (P_SX [i] + P_EX [i]) / 2. Here, i is an ID number when a plurality of pedestrians are extracted.

  Next, step S602 and subsequent steps are repeated until the loop control variable g reaches G in step S605. A pedestrian can be detected by the above processing.

  As described above, detection delay is achieved by optimally selecting an image to be calculated for optical flow using images captured at a fixed period and images captured by adjusting the capture timing according to the vehicle speed. Even when the vehicle speed is low, it is possible to obtain an effect such that a distant object can be detected.

<Second Embodiment>
Next, a second embodiment of the vehicle external environment recognition device of the present invention will be described below with reference to the drawings. This embodiment corresponds to claim 3.

  FIG. 7 is a block diagram showing the configuration of the vehicle external environment recognition device 2000 according to the second embodiment. In the following description, only the parts different from the vehicle external environment recognition device 1000 in the first embodiment will be described in detail, and the same parts will be denoted by the same reference numerals and detailed description thereof will be omitted. What is characteristic in the present embodiment is that the vehicle behavior acquisition unit 2021 takes into account the yaw rate.

  The vehicle external environment recognition device 2000 is incorporated in a camera device mounted on an automobile, an integrated controller, or the like, and is for detecting an object from an image captured by the camera 1010. In the present embodiment, A vehicle or a pedestrian is detected from an image obtained by imaging the front of the host vehicle.

  The vehicle external environment recognition apparatus 2000 is configured by a computer having a CPU, a memory, an I / O, and the like, programmed with predetermined processes and procedures, and repeatedly executes processes at a predetermined cycle T.

  As shown in FIG. 7, the vehicle external environment recognition device 2000 includes an image acquisition unit 1011, a host vehicle behavior acquisition unit 2021, an imaging timing adjustment unit 2031, an optical flow calculation unit 1041, a foreground / background separation unit 2051, An object detection unit 1061.

  The own vehicle behavior acquisition unit 2021 acquires a detection signal of a vehicle speed measurement device mounted on the own vehicle, and acquires the own vehicle speed VSP. Furthermore, the detection signal of the skid prevention device mounted on the own vehicle is acquired, and the yaw rate YR is acquired. Alternatively, the yaw rate YR may be calculated by obtaining a steering angle Str by obtaining a detection signal of the steering device. The own vehicle speed VSP and the yaw rate YR may be acquired by directly inputting each signal to the vehicle external environment recognition apparatus 1000, or may be acquired by performing communication using a LAN (Local Area Network). Good.

  The imaging timing adjustment unit 2031 adjusts the imaging timing of the image acquisition unit using the host vehicle speed VSP and the yaw rate YR acquired by the host vehicle behavior acquisition unit 2021. Specifically, it has a function of setting an imaging timing for capturing a fixed-period acquired image and an irregularly acquired image. Details of the processing will be described later.

  The foreground / background separation unit 2051 analyzes the optical flows IMGOFX [x] [y] and IMGOFY [x] [y], and separates them into the foreground and the background. In the present embodiment, the foreground areas on the image are grouped and expressed as FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g]. Here, g is an ID number when a plurality of foregrounds are extracted. Details of the processing will be described later. The other configuration is the same as that of the vehicle external environment recognition apparatus 1000 already described.

[Imaging timing adjustment unit]
Next, the contents of processing in the imaging timing adjustment unit 2031 will be described.
The imaging timing adjustment unit 2031 determines the next imaging timing T1 and the next imaging timing T2. Here, the imaging timing T2 is determined so that an object on the image captured at the imaging timing T1 moves by a predetermined amount when it appears on the image captured at a fixed period T. For example, when T2 is determined so that all stationary objects move P meters in the front-rear direction, considering the yaw rate, for example, VY = VSPcos (YR) is used, and T2 = P / VY. T1 is obtained from T1 = T−T2 using the processing cycle T. Note that thresholds T2Min and T2Max are provided in order to cope with the case where VSP is low and T2> T, or when VSP is high and T2 exceeds the limit of the capturing speed of the image sensor, and when T2> T2Max, T2 = In the case of T2Max and T2 <T2Min, a process for setting T2 = T2Min is necessary.

[Foreground / Background Separator 2051]
Next, the contents of processing in the foreground / background separation unit 2051 will be described. Since only the step S302 is different from the foreground / background separation unit 1051 in the processing flow, only step S302 will be described here, and the rest will be omitted.

  In step S302, the position of the point (xn, yn) on the latest captured image IMGSRC [x] [y] is predicted on the image T2 seconds before, and a prediction flow is calculated. In the present embodiment, the prediction flow is obtained by the following procedure. First, from the vanishing point and the camera geometry, the coordinate (xn, yn) is assumed to be a point on the road surface, and the world coordinate (Xn, Yn, 0) is obtained. From the own vehicle speed VSP and the capture timing T2, the movement amount PY = VSPcos. (YR) × T2, PX = VSPsin (YR) is calculated to obtain the predicted position (Xn + PX, Yn + PY, 0), and the image coordinate position (xnp, ynp) is calculated. The prediction flow is PFX = (xnp−xn), PFY = (ynp−yn). As described above, taking into account the yaw rate, it is possible to cope with turning.

<Third Embodiment>
Next, a third embodiment of the vehicle external environment recognition device of the present invention will be described below with reference to the drawings. This embodiment corresponds to claims 7, 8, and 9.

  FIG. 8 is a block diagram showing a configuration of a vehicle external environment recognition device 3000 according to the third embodiment. In the following description, only parts different from the vehicle external environment recognition devices 1000 and 2000 in the first embodiment described above will be described in detail, and the same parts will be denoted by the same reference numerals and detailed description thereof will be given. Omitted.

  What is characteristic in the present embodiment is that, instead of optical flow generation, a vanishing point acquisition unit 3041 and a predicted image generation unit 3042 enter, and the foreground / background separation unit 3051 uses the result of the predicted image generation unit 3042 to detect the foreground and background. Is that they are separated.

  The vehicle external environment recognition device 3000 is incorporated in a camera device mounted on an automobile, an integrated controller, or the like, and is for detecting an object from an image captured by the camera 1010. In the present embodiment, It is configured to detect a vehicle or a pedestrian.

  The vehicle external environment recognition device 3000 is configured by a computer having a CPU, a memory, an I / O, and the like, programmed with predetermined processes and procedures, and repeatedly executes processes at predetermined intervals.

  As shown in FIG. 8, the vehicle external environment recognition device 3000 includes an image acquisition unit 1011, a host vehicle behavior acquisition unit 1021, an imaging timing adjustment unit 1031, a vanishing point acquisition unit 3041, a predicted image generation unit 3042, A foreground / background separation unit 3051 and an object detection unit 1061;

  The vanishing point acquisition unit 3041 is a means for acquiring the vanishing point (VX, VY) of the camera 1010. The initial value set in advance or the lane recognition is simultaneously operated in the camera, and a value obtained from the result is obtained. refer. Regarding the lane recognition, the explanation is omitted because it deviates from the main point of the present invention.

  The predicted image generation unit 3042 converts the irregularly acquired image IMGSRC_sub [x] [y] into the fixed period acquired image IMGSRC [x] [y] using the vanishing point (VX, VY), the host vehicle speed VSP, and the camera geometry. The resulting image is converted to generate a converted image IMGSRC ′ [x] [y]. Here, the yaw rate YR may be used. Detailed description will be given later.

  The foreground / background separator 3051 compares the fixed-period acquired image IMGSRC [x] [y] with the converted image IMGSRC ′ [x] [y] and separates the foreground and the background. In the present embodiment, foreground regions on the image are grouped and expressed as FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g]. Here, g is an ID number when a plurality of foregrounds are extracted. Details of the processing will be described later.

[Predicted image generation unit 3042]
Next, the contents of processing in the predicted image generation unit 3042 will be described with reference to FIG. FIG. 9 is a flowchart showing the flow of processing of the predicted image generation unit 3042. First, in step S901, control variable initialization x = 0 and y = 0 for accessing the predicted image IMGSRC ′ [x] [y] is performed.

  Next, in step S902, it is predicted from which position of the irregularly acquired image IMGSRC_sub the point (x, y) of the predicted image IMGSRC ′ [x] [y] has come. The prediction is performed on the assumption that all points on the irregularly captured image IMGSRC_sub [x] [y] are present on the road surface or on a plane perpendicular to the road surface at a predetermined distance. For example, assuming that all exist on the road surface, the point (x, y) of the predicted image IMGSRC ′ [x] [y] is converted to the world coordinates (X, Y, 0) using the vanishing point (VX, VY) and the camera geometry. ), The predicted position P = VSP × T2 is obtained from the capture period T2 and the own vehicle speed VSP, and the world coordinates (X, Y−P, 0) are returned to the image coordinates (x ′, y ′). .

  Note that the yaw rate YR may be used in step S902. In this case, the point (x, y) of the predicted image IMGSRC ′ [x] [y] is converted into the world coordinates (X, Y, 0) using the vanishing point (VX, VY) and the camera geometry, and the capture period T2 The predicted positions PY = T2 × VSPcos (YR) and PX = T2 × VSPsin (YR) are obtained from the vehicle speed VSP and the yaw rate YR, and the world coordinates (X−PX, Y−PY, 0) are obtained as image coordinates (x ′, It can be estimated by returning to y ′).

  Next, in step S903, the coordinates (x ′, y ′) point of the irregularly captured image IMGSRC_sub and the information of IMGSRC_sub [x ′] [y ′] are substituted into IMGSRC ′ [x] [y]. When the coordinates (x ′, y ′) are not integers, the nearest neighbor value may be used, or bilinear interpolation may be performed. Next, in step S904, it is determined whether or not necessary pixel access has been completed. If not completed, the processing from step S902 is repeated, and if completed, the processing ends.

[Foreground / Background Separation Unit 3051]
Next, the contents of processing in the foreground / background separation unit 3051 will be described with reference to FIG. FIG. 10 is a flowchart showing the processing flow of the foreground / background separation unit 3051. First, in step S1001, a difference image IMGSUB between the fixed period acquired image IMGSRC and the predicted image IMGSRC ′ is generated. In the present embodiment, an image obtained by converting a difference for each pixel value into an absolute value is imaged.

  Next, in step S1002, control variables for pixel access of the difference image IMGSUB are initialized (x = 0, y = 0).

  Next, in step S1003, if the point IMGSUB [x] [y] on the difference image is larger than the threshold value, it is determined to be the foreground in step S1004, and if it is less than the threshold value, the background is determined in step S1005. .

  Next, in step S1006, it is determined whether all pixel accesses are completed. If not completed, step S1003 and subsequent steps are repeated.

Next, in step S1007, the determined foreground areas are grouped to obtain foreground areas FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g]. The grouping is performed as follows, for example. First, a binary image is generated with the point determined to be the foreground being 1 and the other points being 0. Next, adjacent points on the two-dimensional map are grouped. If the upper, lower, left, and right ends of each group on the two-dimensional map are GR_SX [g], GR_SY [g], GR_EX [g], GR_EY [g], the foreground region is obtained by the following equation.
FG_SX [g] = GR_SX [g]
FG_SY [g] = GR_SY [g]
FG_EX [g] = GR_EX [g]
FG_EY [g] = GR_EY [g]

  Through the above processing, the foreground areas FG_SX [g], FG_SY [g], FG_EX [g], and FG_EY [g] are obtained.

  As described above, the foreground can be extracted from the difference between the image IMGSRC ′ [x] [y] obtained by converting the irregularly acquired image IMGSRC_sub [x] [y] and the fixed period acquired image IMGSRC [x] [y]. it can.

<Fourth embodiment>
Next, a fourth embodiment of the vehicle external environment recognition device of the present invention will be described below with reference to the drawings. This embodiment corresponds to claim 10.

  FIG. 11 is a block diagram showing a configuration of a vehicle external environment recognition device 4000 according to the fourth embodiment. In the following description, only the parts different from the above-described vehicle external environment recognition devices 1000, 2000, and 3000 will be described in detail, and the same parts will be denoted by the same reference numerals and detailed description thereof will be omitted. What is characteristic in the present embodiment is that a plurality of indefinite period images are captured, and a predicted image is generated from the plurality of images by the predicted image generation unit 3042.

  The vehicle external environment recognition device 4000 is incorporated in a camera device mounted on an automobile, an integrated controller, or the like, and is for detecting an object from within an image captured by the camera 1010. In the present embodiment, It is configured to detect a vehicle or a pedestrian.

  The vehicle external environment recognition device 4000 is configured by a computer having a CPU, a memory, an I / O, and the like, programmed with predetermined processes and procedures, and repeatedly executes processes at a predetermined cycle T.

As shown in FIG. 8, the vehicle external environment recognition device 4000 includes an image acquisition unit 4011, a host vehicle behavior acquisition unit 1021, an imaging timing adjustment unit 4031, a vanishing point acquisition unit 3041, a predicted image generation unit 4042, A foreground / background separation unit 3051 and an object detection unit 1061;
The image acquisition unit 4011 takes in image data obtained by photographing the front of the host vehicle from a camera 1010 attached to a position where the front of the host vehicle can be imaged, and obtains a fixed period acquired image IMGSRC [x] [y] and an irregularly acquired image. IMGSRC_sub1 [x] [y] and IMGSRC_sub2 [x] [y] are stored on the RAM 1020.

  The imaging timing adjustment unit 4031 uses the host vehicle speed VSP acquired by the host vehicle behavior acquisition unit 1021 to adjust the imaging timing of the image acquisition unit. Specifically, it has a function of setting an imaging timing for capturing a fixed-period acquired image and an irregularly acquired image. Details of the processing will be described later.

  The predicted image generation unit 4042 uses the irregularly acquired image IMGSRC_sub [x] [y] as the fixed period acquired image IMGSRC [x] [y] using the vanishing point (VX, VY), the host vehicle speed VSP, and the camera geometry. The resulting image is converted to generate a converted image IMGSRC ′ [x] [y]. Here, the yaw rate YR may be used.

[Imaging timing adjustment unit]
Next, contents of processing in the imaging timing adjustment unit 4031 will be described with reference to FIG. FIG. 12 is a timing chart for explaining the imaging timing of the image acquisition unit 4011.

  The imaging timing adjustment unit 4031 determines the next imaging timing T1 and the next two imaging timings T21 and T22. Here, the imaging timings T21 and T22 are determined so that an object on the image captured at the imaging timing T1 moves by a predetermined amount when it appears on the image captured at a fixed period T. For example, when T21 is determined so that all stationary objects move by P1 meters, it can be determined from T21 = P1 / VSP. Similarly, T22 = P2 / VSP is determined so that the stationary object moves by P2 meters.

  T1 is obtained from T1 = T− (T22) using the processing cycle T. It should be noted that thresholds T2Min and T2Max are provided in order to cope with the case where VSP is low and T22> T, or VSP is high and T1 exceeds the limit of the image pickup device capture speed. When T22> T2Max, T22 = In the case of T2Max and T21 <T2Min, a process for setting T21 = T2Min is necessary.

[Predicted image generation unit 4042]
Next, the contents of processing in the predicted image generation unit 4042 will be described with reference to FIG. FIG. 13 is a flowchart showing a process flow of the predicted image generation unit 4042. First, in step S1301, initialization of control variables x = 0 and y = 0 for accessing the predicted image IMGSRC ′ [x] [y] is performed.

  Next, in step S1302, an image that brings the value of the predicted image IMGSRC ′ [x] [y] is determined. Since the movement near the vanishing point is small, the image IMGSRC_sub2 captured at the long cycle T22 is used, and the image IMGSRC_sub1 captured at the short cycle T21 is used near the vehicle lower end. Specifically, coordinates (x, y) are converted into world coordinates and determined at positions on the world coordinates, or a map is previously stored on the image coordinates, and which image is used according to the map. To do. In this embodiment, it is assumed that all points are present on the road surface, and the point (x, y) is converted to the world coordinates (X, Y, 0) using the vanishing point (VX, VY) and the camera geometry. If the world coordinate Y is greater than or equal to a predetermined value, the image IMGSRC_sub2 is used, and if it is less than the predetermined value, the image IMGSRC_sub1 is used.

  Next, in step S1303, predicted coordinates (x ′, y ′) are calculated. When the image IMGSRC_sub1 is selected in step S1302, the predicted position P1 = VSP × T21 is obtained from the capture period T21 and the vehicle speed VSP with respect to the world coordinates (X, Y, 0) obtained in step S1302, and the world coordinates are obtained. It can be estimated by returning (X, Y-P1, 0) to the image coordinates (x ′, y ′). Similarly, when the image IMGSRC_sub2 is selected, the predicted position P2 = VSP × T22 is obtained from the capture period T22 and the vehicle speed VSP with respect to the world coordinates (X, Y, 0), and the world coordinates (X, Y−P2, It can be estimated by returning (0) to the image coordinates (x ′, y ′).

  Note that the yaw rate YR may be used in step S1303. In this case, in the capture period T21, the predicted position PY1 = T21 × VSPcos (YR) and PX1 = T21 × VSPsin (YR) are obtained from the own vehicle speed VSP and the yaw rate YR, and the world coordinates (X-PX1, Y-PY1,0) are obtained. ) To the image coordinates (x ′, y ′). The same applies to the capturing period T22.

  Next, in step S1304, a predicted image is generated. If the image IMGSRC_sub1 is selected in step S1302, the coordinates (x ′, y ′) point of the irregularly captured image IMGSRC_sub1 and the information of IMGSRC_sub [x ′] [y ′] are substituted into IMGSRC ′ [x] [y]. To do. Similarly, when IMGSRC_sub2 is selected, the coordinates (x ', y') point of the irregularly captured image IMGSRC_sub2 and the information of IMGSRC_sub [x '] [y'] are substituted into IMGSRC '[x] [y]. When the coordinates (x ′, y ′) are not integers, the nearest neighbor value may be used, or bilinear interpolation may be performed.

  Next, in step S1305, it is determined whether or not necessary pixel access has been completed. If not completed, the processing from step S1302 is repeated, and if completed, the processing is terminated.

  As described above, it is possible to detect a distant object by predicting a far object from an image having a long capturing period and predicting a nearby object from an image having a short capturing period.

  Hereinafter, the operation method of the system, such as outputting an alarm according to the preceding vehicle determined from the above-described embodiment or automatically controlling the brake, will be described with reference to FIGS. This description corresponds to claims 11, 12, and 13.

  FIG. 14 is a flowchart showing an operation method of the pre-crash safety system. First, in step S141, obstacle information (PYO [i], PXO [i]) is read.

Next, in step S142, the collision prediction time TTC [i] of each detected object is calculated using equation (1). Here, the relative speed VY1 [i] is obtained by pseudo-differentiating the relative distance PY1 [i] of the object.
TTC [i] = PY1 [i] ÷ VY1 [i] (1)
Further, in step S143, the risk level DRECI [i] for each obstacle is calculated.

  Hereinafter, an example of a calculation method of the risk level DRECI [i] for the object X [i] detected by any one of the above-described vehicle external world recognition devices will be described with reference to FIG.

First, a method for estimating a predicted course will be described. As shown in FIG. 15, when the vehicle position is the origin O, the predicted course can be approximated by an arc of a turning radius R passing through the origin O. Here, the turning radius R is expressed by Equation (2) using the steering angle α of the host vehicle, the host vehicle speed Vsp, the stability factor A, the wheel base L, and the steering gear ratio Gs.
R = (1 + AVsp ² ) × (L · Gs / α) (2)

The stability factor is an important value that determines the magnitude of the change depending on the speed of the steady circular turning of the vehicle. As can be seen from the equation (2), the turning radius R changes in proportion to the square of the host vehicle speed Vsp with the stability factor A as a coefficient. Further, the turning radius R can be expressed by Equation (3) using the host vehicle speed Vsp and the yaw rate γ.
R = Vsp / γ (3)

Next, a perpendicular line is drawn from the object X [i] to the center of the predicted course approximated by the arc of the turning radius R to obtain the distance L [i]. Further, the distance L [i] is subtracted from the own vehicle width H, and when this is a negative value, the danger level DRECI [i] = 0, and when the value is a positive value, the danger level DRECI [i] is given by the following equation (4). ] Is calculated.
DRECI [i] = (HL [i]) / H (4)
In addition, the process of step S141-S143 is set as the structure which performs a loop process according to the detected number of objects.

In step S144, an object satisfying the condition of Expression (5) is selected according to the risk degree DRECI [i] calculated in step S143, and the predicted collision time TTC [i] is the smallest among the selected objects. The object iMin is selected.
DRECI [i] ≧ cDRECI # (5)
Here, the predetermined value cDRECI # is a threshold value for determining whether or not the vehicle collides.

Next, in step S145, it is determined whether or not the brake is automatically controlled in accordance with the predicted collision time TTC [iMin] of the selected object k. If Expression (6) is established, the process proceeds to step S146, brake control is executed, and the process is terminated.
On the other hand, if the expression (6) is not established, the process proceeds to step S147.
TTC [iMin] ≦ cTTCBRK # (6)

In step S147, it is determined whether or not the alarm is output in accordance with the predicted collision time TTC [iMin] of the selected object iMin. If Expression (7) is established, the process proceeds to step S148, where an alarm is output and the process is terminated. In addition, when Expression (7) is not established, neither the brake control nor the warning is executed and the process is terminated.
TTC [iMin] ≦ cTTCALM # (7)

  As described above, the alarm and the brake control can be activated according to the degree of risk of an object determined as a preceding vehicle in any one of the above-described vehicle external recognition devices according to the present invention.

  The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

1000 Vehicle external recognition device 1010 Camera (imaging device)
1020 RAM
1011 Image acquisition unit 1021 Own vehicle behavior acquisition unit 1031 Imaging timing adjustment unit 1041 Optical flow calculation unit 1051 Foreground / background separation unit 1061 Object detection unit 2000 Vehicle external environment recognition device 2021 Own vehicle behavior acquisition unit 2031 Imaging timing adjustment unit 2051 Foreground / background separation Unit 3000 vehicle external environment recognition device 3041 vanishing point acquisition unit 3042 prediction image generation unit 3051 foreground / background separation unit 4000 vehicle external environment recognition device 4011 image acquisition unit 4031 imaging timing adjustment unit 4042 prediction image generation unit VSP own vehicle speed YR yaw rate

Claims (12)

Image capturing means for capturing an image from an imaging device such as a camera;
Own vehicle behavior acquisition means for acquiring own vehicle speed;
Imaging timing adjustment means for adjusting the timing of capturing an image from the imaging device;
It said image capturing means may capture images at a fixed period set by the imaging timing adjustment unit, Captures images irregularly during the fixed period,
The vehicle external environment recognition apparatus, wherein the imaging timing adjustment means adjusts the timing of irregularly taking an image in accordance with the own vehicle speed . Image capturing means for capturing an image from an imaging device such as a camera;
And the vehicle behavior acquisition means for acquiring a vehicle speed and yaw rate,
Imaging timing adjustment means for adjusting the timing of capturing an image from the imaging device ;
The image capturing unit captures an image at a constant cycle set by the imaging timing adjustment unit, and captures an image irregularly during the fixed cycle,
The imaging timing adjustment means, wherein you and adjusting the timing of image capture of irregularly in response to vehicle speed and yaw rate, environment recognizing apparatus for a vehicle. The imaging timing adjustment means further vehicle environment recognizing apparatus according to claim 1 or 2, characterized in that to adjust the timing of image capture according to the maximum detection distance of the object. Foreground / background separating means for separating the foreground and the background using the captured image;
Environment recognizing a vehicle according to any one of claims 1 to 3, characterized in that it comprises a and object detecting means for detecting an object where there is potential for conflict from the foreground separated by the foreground background separating unit apparatus. An optical flow calculating means for calculating an optical flow between images;
5. The vehicle external environment recognition apparatus according to claim 4 , wherein the foreground / background separation means separates the foreground and the background based on an optical flow. Vanishing point acquisition means for acquiring a vanishing point;
Using the vanishing point, a prediction image generation unit that converts an image captured irregularly by the own vehicle behavior acquisition unit and generates a converted image, and
5. The vehicle external environment recognition apparatus according to claim 4 , wherein the foreground / background separation unit separates the foreground and the background based on a difference between the converted image and an image captured at a constant period. The vehicle external environment recognition device according to any one of claims 4 to 6 , wherein the object detection unit determines whether or not the foreground is a vehicle. The vehicle external environment recognition apparatus according to any one of claims 4 to 6 , wherein the object detection means determines whether or not the foreground is a pedestrian. Vanishing point acquisition means for acquiring a vanishing point;
Using the vanishing point, having a predicted image generation means for converting a plurality of images irregularly captured by the vehicle behavior acquisition means and generating a plurality of converted images,
5. The vehicle external environment recognition apparatus according to claim 4 , wherein the foreground / background separation unit separates the foreground and the background based on a difference between the converted image and an image captured at a constant period. A vehicle system comprising the external environment recognition device for a vehicle according to any one of claims 4 to 9 ,
A collision risk calculating means for calculating a risk of collision of the vehicle according to the position of the object detected by the object detecting means;
Control means for performing control to avoid a collision of the own vehicle according to the collision risk calculated by the collision risk calculation means;
A vehicle system comprising: The control means calculates a predicted collision time when the host vehicle collides with the object based on the position of the object, compares the calculated predicted collision time with a preset braking threshold, The vehicle system according to claim 10 , wherein when the predicted collision time is equal to or less than the braking threshold, braking control is performed on the host vehicle. The control means calculates a predicted collision time when the host vehicle collides with the object based on the position of the object, compares the calculated predicted collision time with a preset alarm threshold, 11. The vehicle system according to claim 10 , wherein when the collision prediction time is equal to or less than the warning threshold value, warning control is performed on the driver of the host vehicle.

Images