JP5927110B2 - Vehicle external recognition device - Google Patents

Description

  The present invention relates to a vehicle external environment recognition device for an automobile that implements an alarm, an automatic brake, or the like for avoiding a collision with an obstacle based on information from an image sensor such as a camera.

In order to reduce the number of casualties due to traffic accidents, the development of preventive safety systems to prevent accidents is in progress. A preventive safety system is a system that operates in a situation where there is a high possibility of an accident. For example, when a collision with an obstacle in front of the host vehicle occurs, the driver is alerted, and a collision occurs. A pre-crash safety system has been put into practical use that reduces the damage to passengers by automatic braking when it becomes inevitable.
Here, it is considered that the warning and the automatic brake for the obstacle are preferably performed at an earlier timing when the driver is not aware of the obstacle than when the driver is aware of the obstacle. It is done. Hereinafter, in this document, an indicator relating to the driver's awareness is referred to as “visibility”. Since it is difficult to directly sense visibility from an electroencephalogram or the like, it is necessary to indirectly estimate this in order to provide an application based on visibility.

  In Patent Document 1, a distance to an obstacle is detected by an obstacle sensor such as a camera or a radar, and it is determined whether or not the obstacle is within a radiation temperature range corresponding to a human by an infrared light camera. If the object is a human and the distance to the obstacle is less than or equal to the predetermined distance, turn on the horn and increase the light distribution by directing the optical axis of the headlamp in the direction of the obstacle, and In addition, it is described that visibility is determined based on fog lamp on / off and a wiper speed, and the predetermined distance is increased as visibility decreases.

  In Patent Document 2, the degree of danger of the object is judged based on the lateral movement speed, position and size of the forward moving object, and the driver's attention is alerted when the degree of danger is greater than a predetermined value. And making it easier to perform the alarm by correcting the predetermined value to be small during rainy weather or nighttime when visibility is poor.

  In Patent Document 3, a driver's line of sight is detected, a pedestrian is detected with a far-infrared camera, and the brightness and background of the image at the position of the pedestrian are compared with a visible light camera. It is described that it is determined that the visibility is poor, and in the case where the visibility is poor, a bright round frame is displayed at the position of the viewpoint corresponding to the position of the pedestrian.

JP 2001-91618 A JP 2000-251200 A JP 2005-135037 A

  However, the techniques described in Patent Documents 1 and 2 have a problem in that the visibility is uniformly poor during rainy weather or nighttime, and the visibility for each obstacle is not considered. For example, if a pedestrian in front of the host vehicle is wearing bright clothes, the driver is likely to notice the pedestrian even when it is raining or at night. If the warning is given early in such a case, the driver may be bothered or disturbed.

  On the other hand, the technique described in Patent Document 3 compares the brightness of the image at the position of the pedestrian (obstacle) with the background, and it seems that the visibility for each obstacle is taken into consideration. However, depending on the color of the pedestrian's clothes, the driver may be able to easily recognize the pedestrian even if the difference between the brightness of the image at the pedestrian's position and the background is small. Even if the difference between the brightness of the image at the person's position and the background is large, the driver may not be able to notice the pedestrian. Therefore, it cannot be said that the visibility of obstacles is accurately determined, and it is considered that there is no particular problem if the display is simply changed. There is room for improvement in terms of efficient implementation and suppression of inconvenience to the driver.

  Therefore, the present invention improves the accuracy of estimating the ease of recognition of individual obstacles (hereinafter referred to as “saliency” in this document), the obstacles that the driver is not likely to notice, and the driver It is an object of the present invention to provide a vehicle external recognition device capable of changing the timing of obstacle avoidance control in a vehicle such as an alarm or an automatic brake with an obstacle that is likely to be noticed or highly likely to be noticed.

Therefore, an external environment recognition device for a vehicle according to one aspect of the present invention includes an image acquisition unit that acquires an image around the host vehicle, an information acquisition unit that acquires position information of an obstacle around the host vehicle, and the obstacle in the image. Based on the luminance component of the predetermined area based on the color information of the predetermined area including the obstacle, based on the obstacle saliency estimation unit that estimates the saliency of the obstacle, and based on the estimated saliency of the obstacle, to have a, and the vehicle control determination unit determines the output necessity or the output timing of the execution command of the obstacle avoidance control in a vehicle. The obstacle saliency estimating unit calculates an image saliency of an obstacle using a saliency map based on at least one of a luminance component, a color component, and an edge angle component of the predetermined region of the image, The saliency of the obstacle is estimated based on the calculation result. The vehicle control determination unit calculates a collision time until the host vehicle collides with the obstacle, and outputs an execution command of the obstacle avoidance control when the collision time is equal to or less than a threshold, The higher the saliency, the smaller the threshold is set.

According to another aspect of the present invention, there is provided an external environment recognition apparatus for a vehicle, wherein an image acquisition unit that acquires an image around the host vehicle and a luminance component based on color information of the entire image are used to recognize each pixel of the image. A pixel saliency estimating unit for estimating the position, a position information acquiring unit for acquiring position information of an obstacle around the host vehicle, and the obstacle based on the saliency of each pixel at the position of the obstacle in the image. An obstacle saliency estimating unit that estimates the saliency of the vehicle, and a vehicle control determination unit that determines whether or not an execution command for executing obstacle avoidance control in the vehicle is output based on the estimated saliency of the obstacle. And the pixel saliency estimation unit uses the saliency map based on at least one of the luminance component, the color component, and the edge angle component of the entire image, and the image saliency of each pixel of the image. Calculate gender. The vehicle external environment recognition apparatus further includes an obstacle detection unit capable of detecting the obstacle by image processing from the image, and the obstacle detection unit is calculated based on the saliency of each pixel of the image. In addition, an obstacle is detected from the low-saliency area of the image, and the position information of the obstacle is output to the position information acquisition unit.

  The vehicle external environment recognition device accurately estimates the saliency of obstacles around the host vehicle compared to the prior art, and is likely to be noticed by the driver and obstacles that are likely not noticed by the driver. It is possible to change the timing of obstacle avoidance control in a vehicle such as an alarm or an automatic brake with an obstacle that is likely to be present.

It is a block diagram which shows the structure of the external field recognition apparatus for vehicles by 1st Embodiment of this invention. It is a flowchart which shows an example of the process in an obstacle saliency estimation part. It is a flowchart which shows the example of calculation of the image saliency of an obstruction using a saliency map. It is a flowchart which shows the other example of the process in an obstacle saliency estimation part. It is a flowchart which shows an example of the process in a vehicle control determination part. It is a block diagram which shows the structure of the external field recognition apparatus for vehicles by 2nd Embodiment of this invention. It is a block diagram which shows the structure of the external field recognition apparatus for vehicles by 3rd Embodiment of this invention. It is a block diagram which shows the structure of the external field recognition apparatus for vehicles by 4th Embodiment of this invention. It is a flowchart which shows an example of the process in an obstruction detection part. It is a figure which shows the flow of calculation of the image saliency using a saliency map.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.
<First Embodiment>
FIG. 1 is a block diagram showing the configuration of a vehicle external environment recognition apparatus 1000 according to the first embodiment of the present invention. The vehicle external environment recognition device 1000 is incorporated in a camera device or an integrated controller mounted on an automobile and detects an object from an image captured by the camera 1010 of the camera device. It is configured to detect obstacles around the host vehicle.

  The vehicle external environment recognition apparatus 1000 is configured by a computer having a CPU, a memory, an I / O, and the like. A predetermined process is programmed and the process is repeatedly executed at a predetermined cycle T.

  As shown in FIG. 1, the vehicle external environment recognition apparatus 1000 includes an image acquisition unit 1011, an obstacle information acquisition unit 1021, an obstacle saliency estimation unit 1031, and a vehicle control determination unit 1041.

  The image acquisition unit 1011 acquires an image IMGSRC [x] [y] [c] obtained by capturing the surroundings of the host vehicle from a camera 1010 attached to a position where the front of the host vehicle can be imaged, and stores the acquired image on the RAM. Here, the image IMGSRC [x] [y] [c] is a two-dimensional array, x and y indicate image coordinates, and c indicates R, G, and B components (color information).

  The obstacle information acquisition unit 1021 acquires obstacle position information PX [b] and PY [b] from a device capable of detecting the presence and position of an obstacle around the host vehicle such as a sonar, radar, or stereo camera. To do. Here, PX indicates the position in the left-right direction of the host vehicle, and PY indicates the position in the front-rear direction of the host vehicle. Further, b is the ID number of each obstacle when a plurality of obstacles are detected. The obstacle information acquisition unit 1021 may acquire obstacle position information PX [b], PY [b] by directly inputting a signal from the device, or using a LAN (Local Area Network). You may acquire by performing communication.

The obstacle saliency estimating unit 1031 inputs the image IMGSRC [x] [y] [c] and the obstacle position information PX [b] and PY [b], and each obstacle is based on the input image and position information. Estimate the saliency VSBLT [b] of the object. Details of the processing in the obstacle saliency estimating unit 1031 will be described later.
Note that the saliency in this embodiment is obtained by quantifying the visibility of a driver using information obtained from a camera image, and is not a value obtained by directly measuring the saliency of an actual driver from an electroencephalogram. . In this embodiment, for example, an indicator is adopted that is high for a pedestrian wearing a light color in a dark background and low for a pedestrian wearing a black background in a dark background. A specific calculation method will be described later.

  The vehicle control determination unit 1041 inputs the saliency VSBLT [b] of each obstacle and the position information PX [b], PY [b] of the obstacle, determines whether or not the obstacle avoidance control in the vehicle is necessary, When obstacle avoidance control is necessary, the output timing of the execution command is determined. Here, the obstacle avoidance control in the vehicle includes not only control for avoiding a collision between the host vehicle and the obstacle, but also control for reducing damage caused by the collision with the obstacle. Inform the driver of the presence of obstacles that may collide with the vehicle, make pedestrians, etc., as obstacles recognize the approach of the vehicle, avoid or collide with obstacles by automatic braking This includes reducing the damage caused by Details of the processing in the vehicle control determination unit 1041 will be described later.

[Obstacle saliency estimation unit]
The contents of processing in the obstacle saliency estimation unit 1031 will be described with reference to FIGS. FIG. 2 is a flowchart showing the calculation process of the obstacle saliency VSBLT [b] performed by the obstacle saliency estimation unit 1031.

In FIG. 2, in step S201, initialization is performed with b = 0.
In step S202, from the obstacle position information PX [b], PY [b] acquired from the obstacle information acquisition unit 1021, an area corresponding to the obstacle on the image IMGSRC [x] [y] [c] ( R [b] (hereinafter referred to as “obstacle region”) is calculated. This calculation is to convert a three-dimensional position in the space to an area on the image. For example, the internal parameters and external parameters of a camera mounted on the vehicle are measured in advance, and the pre-measurement is performed. This is implemented using parameters and geometric information of the camera (such as the installation angle of the camera and the relationship between the distance and the size). The obstacle region R [b] is calculated as, for example, a minimum rectangular region including an obstacle on the image IMGSRC [x] [y] [c].

  In step S203, the image saliency IV [b] of the obstacle is calculated from the obstacle region R [b] and the surrounding region r [b]. The surrounding region r [b] can be, for example, a region that is one pixel or several pixels outside the obstacle region R [b]. A method for calculating the image saliency IV [b] will be described later. The obstacle region R [b] and the surrounding region r [b] correspond to the “predetermined region including the obstacle” in the present invention.

  In step S204, the past image saliency IV_z1 [b],..., IV_zN [b] calculated for each of the obstacles that are currently targeted from one cycle before to N cycles before is acquired. The acquisition of the past image saliency can be realized, for example, by the vehicle external environment recognition apparatus 1000 storing information on each past obstacle up to N cycles before in a RAM or the like.

In step S205, obstacle saliency VSBLT [b] is obtained by the following equation.
VSBLT [b] = MAX (IV [b], IV_z1 [b],..., IV_zN [b])
Even if the image saliency calculated from the current image is low, if the image saliency calculated immediately before or before is high, the driver may be aware of the obstacle. It is considered high. Therefore, in step S205, the obstacle saliency VSBLT [b] is calculated in consideration of not only the image saliency IV [b] calculated from the current image but also the image saliency IV_z [b] calculated in the past. Thereby, it is suppressed that an image saliency is calculated low about an obstacle with a high possibility that the driver is aware of the obstacle.

  In step S206, b is incremented. If b is less than the number of obstacles B, the process returns to step S202. If it is greater than or equal to the number of obstacles B, all obstacles have been checked, and the process is terminated.

Next, calculation of image saliency IV [b] performed in step S203 will be described.
In the present embodiment, the image saliency IV [b] is calculated based on the luminance components of the obstacle region R [b] and the surrounding region r [b]. For simplicity, the average value (including the weighted average value) of the color information (RGB components) of each pixel of the image IMGSRC [x] [y] [c] is used as the luminance value of each pixel, and the obstacle region R [b] The difference between the average value of the luminance values of the respective pixels and the average value of the luminance values of the respective pixels in the surrounding region r [b] can be set as the image saliency IV [b].

  However, in this embodiment, the image saliency IV [b] is calculated using a technique called saliency map. The saliency map is a model for calculating a region that easily attracts human visual attention in an image, and is a well-known technique described in papers and the like. Therefore, a detailed description of the saliency map is omitted here, and the flow of calculating the image saliency IV [b] using the saliency map will be described.

FIG. 3 shows a flow of image saliency IV [b] using a saliency map, and FIG. 10 shows a flow of calculation of image saliency IV [b] using a saliency map.
In step S301, a partial image IMGSRC_R [x] [y] [c] [b] is generated from the obstacle region R [b] and the surrounding region r [b] on the image IMGSRC [x] [y] [c]. Nine images obtained by extracting the luminance component, the color component, and the edge angle component (direction component) from the partial image are generated. In the present embodiment, the luminance component is an average value image I in which the average value of the RGB components of each pixel is the luminance value of each pixel, and the color components are R, G, B, and Y calculated from the RGB components. Is one ingredient. Also, four components obtained by applying Gabor filters of 0 [deg], 45 [deg], 90 [deg], and 135 [deg] to the luminance component are generated as edge angle components.

  In step S302, nine partial images having nine components (luminance component, four color components, and four edge angle components) generated from the partial image IMGSRC_R [x] [y] [c] [b] as pixel values. Generate an image pyramid (Gaussian pyramid). The image pyramid is configured as a set of images generated at each stage when an operation of blurring an image and reducing the resolution to half is repeated. In this embodiment, a nine-stage image pyramid is generated. Therefore, since nine stages of image pyramids are generated from nine partial images, a total of 81 images are generated at this point.

  In step S303, two images having different scales in one image pyramid are selected, the smaller image is enlarged to the larger image size, and a difference image between them is obtained.

As for the luminance component, two images having different scales are selected, and the luminance component difference image FI is obtained by the following equation.
FI (c, s) = | GI (c) −GI (s) |
Here, c and s are the number of stages of the image pyramid, and s is larger than c. In other words, the image of the number of stages s is smaller than the image of the number of stages c.

For the color component image, a difference image between R and G, B and Y is obtained by the following equation.
FRG (c, s) = | (GR (c) −GG (c)) − (GG (s) −GR (s)) |
FBY (c, s) = | (GB (c) −GY (c)) − (GY (s) −GB (s)) |

  As for the edge angle component, similarly to the luminance component, a difference between corresponding pixel values is calculated between images of different stages in each of the pyramid images of 0 [deg], 45 [deg], 90 [deg], and 135 [deg]. To obtain a difference image.

  Here, (c = 2, s = 5), (c = 2, s = 6), (c = 3, s = 6), (c = 3, s = 7), (c = 4, Difference images are calculated for six types (s = 7) and (c = 4, s = 8). As a result, a total of 42 difference images are generated: 6 for the luminance component, 12 for the color component, and 24 for the edge angle component.

  In step S304, 6 difference images of luminance components, 12 difference images of color components, and 24 difference images of edge angle components are normalized and overlaid, and a luminance component feature map FMI [x] [y] ] [B], a color component feature map FMC [x] [y] [b], and an edge angle component feature map FMO [x] [y] [b].

In step S305, each component is normalized to create a saliency map SM [x] [y] [b] from the weighted average.
SM [x] [y] [b] = k_I × FMI [x] [y] [b] + k_C × FMC [x] [y] [b] + k_O × FMO [x] [y] [b]
Here, k_I, k_C, and k_O are weighting factors, and are all set to 0.33 in this embodiment.
FIG. 10 shows the output at each stage in the processing related to the generation of the saliency map described above. The finally obtained saliency map SM [x] [y] [b] is a portion obtained by cutting out the obstacle region R [b] and the surrounding region r [b] from the image IMGSRC [x] [y] [c]. The image is the same size as the image IMGSRC_R [x] [y] [c] [b], and the pixel value of each coordinate (x, y) represents the saliency at that coordinate.

In step S306, the image saliency IV [b] is calculated using the values in the obstacle region R [b] of the saliency map SM [x] [y] [b] created as described above. To do. In this embodiment, the maximum value of the pixel values in the obstacle region R [b] in the saliency map SM [x] [y] [b] is extracted and is used as the image saliency IV of the obstacle region R [b]. [B].
The image saliency IV [b] of the obstacle can be calculated by the arithmetic processing as described above.

  In this way, by using the saliency map, the image saliency IV [b] can be calculated in consideration of the ease of attracting human visual attention, so the estimation accuracy of the saliency of the obstacle for the driver is calculated. Can be increased.

  In creating the saliency map SM [x] [y] [b] described above, the number of image pyramid stages is selected according to the size of the obstacle region R [b]. May be. Further, here, a feature map of each component is created by extracting three components of a luminance component, a color component, and an edge angle component from the obstacle region R [b] and the surrounding region r [b]. However, the present invention is not limited to this, and the generation of the saliency map SM and the image saliency IV [b] are calculated based on the feature map of at least one of the luminance component, the color component, and the edge angle component. Also good.

  Further, the obstacle saliency estimating unit 1031 may estimate the saliency of the obstacle in consideration of information other than information obtained from the image. Here, as an example, a method of changing the saliency of an obstacle calculated from an image using a driver's blind spot position acquired in advance will be described. The driver's blind spot is, for example, an average of the positions of the eyes of the driver based on a plurality of seated states in the driver's seat, and from there, an area that is shaded by the pillar of the vehicle and / or an area that is not visible by the mirror Is obtained in advance by calculating which region is in world coordinates.

FIG. 4 is a flowchart showing a calculation process of obstacle saliency VSBLT [b] in consideration of the driver's blind spot.
In FIG. 4, steps S401 to S405 are the same as steps S201 to S205 in FIG.

  In step S406, it is determined whether an obstacle is present in the driver's blind spot, that is, whether the obstacle positions PX [b], PY [b] are the driver's blind spot positions. If the obstacle position PX [b], PY [b] is the driver's blind spot position, the process proceeds to step S407, and if the obstacle position is not the driver's blind spot position, the process proceeds to step S408.

In step S407, the predetermined value q is subtracted from the obstacle saliency VSBLT [b] calculated in step S405.
In step S408, b is incremented. If b is less than the number of obstacles B, the process returns to step S402. If b is greater than or equal to the number of obstacles B, all objects have been checked, and the process ends.

  Thus, when an obstacle is present in the driver's blind spot, even if the saliency calculated from the image is high, it is estimated that the saliency is not visible to the driver, and the saliency is set low. Thereby, even if the saliency on the image is high, the obstacle can be handled as low saliency.

[Vehicle control determination unit]
The content of the process in the vehicle control determination part 1041 is demonstrated using FIG.
FIG. 5 is a flowchart illustrating an example of processing performed in the vehicle control determination unit 1041.

In FIG. 5, in step S501, initialization is performed with b = 0.
In step S502, it is determined whether or not the vector (speed vector) of the relative speeds VX [b] and VY [b] of the obstacle viewed from the own vehicle is directed toward the own vehicle. The relative speeds VX [b] and VY [b] of the obstacles are, for example, the positions PX [b] and PY [b] of the obstacles and the positions PX_z1 [b] and PY_z1 [b] of the obstacles one cycle ago. It can be calculated by the difference approximation. Whether or not the speed vector is directed in the direction of the host vehicle is determined by, for example, considering the position PX [b], PY [b] of the obstacle and the position of the host vehicle when a coordinate system with the center of the host vehicle as the origin is considered. It can be determined by checking whether the direction of the velocity vector is between straight lines connecting the four corners. If the speed vector is facing the direction of the host vehicle, the process proceeds to step S503, and if not, the process proceeds to step S511.

In step S503, the collision time TTC [b] is calculated. The collision time TTC [b] is a parameter that represents how many seconds the host vehicle collides with an obstacle. Here, in order to handle only obstacles that are heading toward the host vehicle, the following equation can be used.
PL [b] = SQRT (PX [b] × PX [b] + PY [b] × PY [b])
VL [b] = SQRT (VX [b] × VX [b] + VY [b] × VY [b])
TTC [b] = PL [b] / VL [b]
Here, SQRT () is a function for calculating the square root.

  In step S504, the saliency VSBLT [b] of the obstacle is compared with the first threshold value THV_H. If the obstacle saliency VSBLT [b]> the first threshold THV_H, the process proceeds to step S508. If the obstacle saliency VSBLT [b] ≦ the first threshold THV_H, the process proceeds to step S505.

  In step S505, the saliency VSBLT [b] of the obstacle is compared with the second threshold value THV_L (<first threshold value THV_H). If the obstacle saliency VSBLT [b]> second threshold THL_L, the process proceeds to step S507. If the obstacle saliency VSBLT [b] ≦ second threshold THL_L, the process proceeds to step S506.

  In steps S506 to S508, a threshold value THT to be compared with the collision time TTC [b] calculated in step S503 is set. That is, in step S506, the threshold value THT = THT_H (upper threshold value) is set, in step S507, the threshold value THT = THT_M (intermediate threshold value) is set, and in step S508, the threshold value THT = THT_L (lower threshold value) is set. Here, THT_H> THT_M> THT_L.

  In step S509, the collision time TTC [b] is compared with the threshold value THT. If the collision time TTC [b]> the threshold value THT, the process proceeds to step S511, and if the collision time TTC [b] ≦ the threshold value THT, the process proceeds to S510.

  In step S510, a flag (alarm activation flag) for operating a vehicle alarm is set to ON. For example, the flag is directly input to an alarm device (not shown) or is transmitted to the alarm device using a LAN (Local Area Network) to activate the alarm.

  In step S511, b is incremented. If b is less than the number of obstacles B, the process returns to step S502. If b is greater than or equal to the number of obstacles B, all objects have been checked, and the process ends.

As described above, the vehicle control determination unit 1041 reduces the threshold THT compared with the collision time TTC [b] when the obstacle saliency VSBLT [b] is high, and the obstacle saliency VSBLT [b]. Is low, the threshold value THT to be compared with the collision time TTC [b] is set to be large, and as a result, the alarm is activated at an earlier timing as the obstacle becomes less noticeable. As a result, a collision with an obstacle can be avoided more reliably, and an alarm can be activated earlier than necessary to prevent the driver from being bothered.
Here, the alarm is activated as the obstacle avoidance control in the vehicle, but it is needless to say that the present invention can be similarly applied to an automatic brake other than the alarm.

  The vehicle external environment recognition apparatus 1000 according to the first embodiment estimates the saliency of an obstacle using a saliency map, and executes an obstacle avoidance control execution command in the vehicle based on the estimation result and the position information of the obstacle. Determine whether output is necessary or output timing. This makes it possible to estimate the saliency of the obstacle in consideration of the ease of attracting human visual attention.For example, the obstacle avoidance in the vehicle from the earlier timing as the obstacle is hard to notice for the driver. While the control is executed, the execution of the obstacle avoidance control can be delayed as much as possible for an obstacle that the driver is likely to notice or is likely to notice. For this reason, the troublesomeness given to the driver can be greatly reduced, and the obstacle avoidance control can be executed appropriately and effectively.

Second Embodiment
Next, a second embodiment of the present invention will be described.
FIG. 6 is a block diagram showing a configuration of a vehicle external environment recognition device 2000 according to the second embodiment of the present invention. In the following description, elements common to the vehicle external environment recognition apparatus 1000 according to the first embodiment are denoted by the same reference numerals and the functions thereof are also the same.

  A feature of the second embodiment is that the vehicle external environment recognition device 2000 includes an obstacle detection unit that can detect an obstacle by image processing from the images IMGSRC [x] [y] [c] acquired by the image acquisition unit. The information (obstacle image position) detected by the obstacle detection unit is output to the obstacle information acquisition unit 2012.

  The vehicle external environment recognition device 2000 is incorporated in a camera device mounted on an automobile, an integrated controller, or the like, and is used to detect an object from an image captured by the camera 1010. An obstacle in front of the vehicle is detected.

  The vehicle external environment recognition apparatus 2000 is configured by a computer having a CPU, a memory, an I / O, and the like. A predetermined process is programmed and the process is repeatedly executed at a predetermined cycle.

  As shown in FIG. 6, the vehicle external environment recognition apparatus 2000 includes an image acquisition unit 1011, an obstacle detection unit 2012, an obstacle information acquisition unit 2021, an obstacle saliency estimation unit 1031, and a vehicle control determination unit 1041. And having.

  The obstacle detection unit 2012 processes the image IMGSRC [x] [y] [c], detects a predetermined obstacle, and outputs the position. In this embodiment, a pedestrian is detected as an obstacle, and the obstacle detection unit 2012 detects a pedestrian from the image IMGSRC [x] [y] [c], and the position of the pedestrian on the image. IX [b] and IY [b] are obtained and output to the obstacle information acquisition unit 2021. In the present embodiment, it is not necessary to add special processing to an algorithm for detecting a pedestrian from an image, and a known algorithm can be applied. Therefore, detailed description thereof is omitted.

The obstacle information acquisition unit 2021 obtains the world coordinates PX [b] and PY [b] of the pedestrian from the positions IX [b] and IY [b] of the pedestrian on the image obtained by the obstacle detection unit 2012. calculate. Such calculation is performed, for example, by camera geometric calculation using internal parameters and external parameters of the camera 1010 that can be measured in advance.
Other processes and the like are the same as those of the vehicle external environment recognition apparatus 1000 according to the first embodiment.

  In the vehicle external environment recognition device 2000 according to the second embodiment, the obstacle detection unit 2012 detects the presence and position of an obstacle such as a pedestrian by processing the images IMGSRC [x] [y] [c]. Thereby, even if there is no device such as a sonar, radar, or stereo camera that can detect the distance to the obstacle, the same effect as the vehicle external environment recognition device 1000 according to the first embodiment can be obtained.

<Third Embodiment>
Next, a third embodiment of the present invention will be described.
FIG. 7 is a block diagram showing a configuration of a vehicle external environment recognition device 3000 according to the third embodiment of the present invention. In the following description, elements common to the vehicle external environment recognition apparatuses 1000 and 2000 according to the first and second embodiments are denoted by the same reference numerals and the functions thereof are also the same.

  A feature of the third embodiment is that the vehicle external environment recognition device 3000 includes a pixel saliency estimation unit that estimates the saliency of each pixel of the image IMGSRC [x] [y] [c]. It is a point that the saliency of the obstacle is estimated from the saliency.

  The vehicle external environment recognition device 3000 is incorporated in a camera device mounted on an automobile, an integrated controller, or the like, and is used for detecting an object from an image captured by the camera 1010. An obstacle in front of the vehicle is detected.

  The vehicle external environment recognition device 3000 is configured by a computer having a CPU, a memory, an I / O, and the like. A predetermined process is programmed, and the process is repeatedly executed at a predetermined cycle.

  As shown in FIG. 7, the vehicle external environment recognition device 3000 includes an image acquisition unit 1011, a pixel saliency estimation unit 3012, an obstacle information acquisition unit 1021, an obstacle saliency estimation unit 3031, and a vehicle control determination unit. 1041.

  The pixel saliency estimation unit 3012 generates a saliency map IMGVSB [x] [y] indicating the saliency of each pixel of the image from the entire image IMGSRC [x] [y] [c]. Specifically, the flowchart shown in FIG. 3 is applied to the entire image, and the saliency map obtained by performing steps S301 to S305 for the entire image is defined as the saliency map IMGVSB [x] [y]. Can do.

  The obstacle saliency estimation unit 3031 uses the saliency map IMGVSB [x] [y] and the obstacle positions PX [b] and PY [b] acquired by the obstacle information acquisition unit 1021 to check the obstacles. The saliency VSBLT [b] is calculated. The obstacle saliency estimating unit 3031 calculates the obstacle saliency VSBLT [b] in substantially the same manner as the flowcharts shown in FIGS. 2 and 4, but the procedure for calculating the image saliency IV [b] is as follows. Different. Specifically, in the first and second embodiments, the obstacle region on the image IMGSRC [x] [y] [c] from the obstacle positions PX [b] and PY [b] in steps S202 and S402. R [b] is calculated, and then image saliency IV [b] is calculated from the obstacle region R [b] and the surrounding region r [b] in steps S203 and S403. That is, the image saliency IV [b] is calculated from IMGSRC [x] [y] [c]. In contrast, in this embodiment, the image saliency IV [b] is not obtained from the image IMGSRC [x] [y] [c], but the saliency map IMGVSB is obtained from the image IMGSRC [x] [y] [c]. [X] [y] is generated, and the image saliency IV [b] is obtained by summing the values in the obstacle region R [b] in the saliency map IMGVSB [x] [y].

  In the vehicle external environment recognition device 3000 according to the third embodiment, the same effects as those of the vehicle external environment recognition devices 1000 and 2000 according to the first and second embodiments can be obtained. In particular, the vehicle external environment recognition device 3000 according to the third embodiment creates a saliency map for the entire image, and estimates the saliency of the obstacle based on the saliency of each pixel in the obstacle region in the saliency map. Therefore, the saliency of the obstacle in the entire image is taken into consideration, and the estimation accuracy of the saliency of the obstacle is further improved.

<Fourth embodiment>
Next, a fourth embodiment of the present invention will be described.
FIG. 8 is a block diagram showing a configuration of a vehicle external environment recognition device 4000 according to the fourth embodiment of the present invention. In addition, in the following description, the same number is attached | subjected about the element which is common in the external field recognition apparatuses 1000, 2000, and 3000 for vehicles by 1st-3rd embodiment, and the function shall also be the same.

  The feature in the fourth embodiment includes an obstacle detection unit that can detect an obstacle from an image by image processing, and the estimation result of the pixel saliency estimation unit that estimates the saliency of each pixel of the entire image This is the point referenced by the obstacle detection unit.

  The vehicle external environment recognition device 4000 is incorporated in a camera device mounted on an automobile, an integrated controller, or the like, and is for detecting an object from an image captured by the camera 1010. It is configured to detect obstacles around the vehicle.

  The vehicle external environment recognition device 4000 is configured by a computer having a CPU, a memory, an I / O, and the like. A predetermined process is programmed, and the process is repeatedly executed at a predetermined cycle.

  As shown in FIG. 8, the vehicle external environment recognition apparatus 4000 includes an image acquisition unit 1011, a pixel saliency estimation unit 3012, an obstacle detection unit 4013, an obstacle information acquisition unit 2021, and an obstacle saliency estimation unit. 3031 and a vehicle control determination unit 1041.

  The obstacle detection unit 4013 dynamically changes the processing using the saliency map IMGVSB [x] [y] generated by the pixel saliency estimation unit 3012 and uses the image IMGSRC [x] [y] [c]. Detect obstacles. Here, it is assumed that the obstacle detection unit 4013 detects a pedestrian as an obstacle, and an example of the processing will be described.

FIG. 9 is a flowchart illustrating an example of processing performed by the obstacle detection unit 4013.
In FIG. 9, in step S901, the window size (w, h) for searching the image IMGSRC [x] [y] [c] is initialized.
In step S902, the search coordinates (x, y) in the image are initialized. In the present embodiment, the upper-left corner coordinates of the window are set as in-image search coordinates (x, y).

  In step S903, the average value (pixel saliency average value) VSB (x, y, w, saliency) of each pixel in the rectangular area determined by the window size (w, h) and the search coordinates (x, y) in the image. h) is calculated. This calculation is performed by referring to the saliency map IMGVSB [x] [y].

  In step S904, the pixel saliency average value VSB (x, y, w, h) is compared with a threshold value THV_D. If the pixel saliency average value VSB (x, y, w, h)> threshold value THV_D, the process proceeds to step S906, and if the pixel saliency average value VSB (x, y, w, h) ≦ threshold value THV_D, step is performed. The process proceeds to S905.

  In step S905, a pedestrian is detected from the image IMGSRC [x] [y] [c] within a rectangular area determined by the window size (w, h) and the in-image search coordinates (x, y). In the present embodiment, a special process is not necessary for the algorithm for detecting a pedestrian, and a known algorithm can be applied, and thus the description thereof is omitted.

  In step S906, the search coordinates (x, y) in the image are updated. In this embodiment, the search coordinates (x, y) in the image are updated by a raster scan method. In raster scanning, x is first advanced by a predetermined number of steps dx from the initial value, and when x + w exceeds the horizontal width of the image, x is returned to the initial value, y is advanced by a predetermined number of steps dy, and x is increased by a predetermined number of steps. The process of advancing by the number of steps dx is repeated. In this embodiment, dx = dy = 1.

  In step S907, it is determined whether or not the intra-image search with the window size (w, h) is completed, that is, whether or not the raster scan of the intra-image search coordinates (x, y) has been completed. If the search within the image is not completed, the process returns to step S903, and if the search within the image is completed, the process proceeds to step S908.

  In step S908, the window size (w, h) is updated. In the present embodiment, a value obtained by multiplying both of the window sizes (w, h) by a predetermined coefficient α is set as a new window size (w, h). In the present embodiment, the predetermined coefficient α = 1.2.

  In step S909, it is determined whether any of the window sizes (w, h) is larger than a predetermined threshold value or image size. If it is not larger, the process returns to step S902. If it is larger, the intra-image search is terminated, and the process is terminated.

  As described above, the obstacle detection unit 4013 is a region with low saliency on the image based on the saliency map IMGVSBL [x] [y] generated from the image IMGSRC [x] [y] [c]. To detect pedestrians. Thereby, when detecting a pedestrian from an image, it is possible to detect a pedestrian that is difficult for the driver to notice while suppressing the number of executions of the pedestrian detection algorithm.

  In the vehicle external environment recognition device 4000 according to the fourth embodiment, the same effects as those of the vehicle external environment recognition device 1000, 2000, 3000 according to the first to third embodiments can be obtained. In particular, according to the vehicle external environment recognition device 4000 according to the fourth embodiment, since a pedestrian is detected from a region with low saliency on the image, a pedestrian that is difficult for the driver to notice can be detected earlier, A collision with a pedestrian can be effectively avoided. The same effect can be obtained for obstacles other than pedestrians by applying an obstacle detection algorithm other than pedestrians instead of or in addition to the pedestrian detection algorithm.

  As mentioned above, although several preferable embodiment of this invention was described, this invention is not limited to each above-mentioned embodiment, A various deformation | transformation and change are possible based on the technical idea of this invention. Of course.

1000 Vehicle external recognition device 1010 Camera 1011 Image acquisition unit 1021 Obstacle information acquisition unit 1031 Obstacle saliency estimation unit 1041 Vehicle control determination unit 2000 Vehicle external recognition device 2012 Obstacle detection unit 2021 Obstacle information acquisition unit 3000 For vehicles External recognition device 3012 Pixel saliency estimation unit 3031 Obstacle saliency estimation unit 4000 External recognition device for vehicle 4013 Obstacle detection unit

Claims (6)

An image acquisition unit for acquiring an image around the host vehicle;
An information acquisition unit for acquiring position information of obstacles around the host vehicle;
An obstacle saliency estimating unit that estimates the saliency of the obstacle based on the luminance component of the predetermined area based on the color information of the predetermined area including the obstacle in the image;
A vehicle control determination unit that determines whether or not output of an execution command for obstacle avoidance control in the vehicle is based on the estimated saliency of the obstacle, and an output timing;
The obstacle saliency estimating unit calculates an image saliency of the obstacle using a saliency map based on at least one of a luminance component, a color component, and an edge angle component of the predetermined region of the image. to estimate the salience of the obstacle on the basis of the calculation results,
The vehicle control determination unit calculates a collision time until the host vehicle collides with the obstacle, and outputs an execution command of the obstacle avoidance control when the collision time is equal to or less than a threshold, The higher the saliency, the smaller the threshold is set.
Vehicle external recognition device. The vehicle external environment recognition device according to claim 1, wherein the vehicle control determination unit calculates the collision time based on a position of the obstacle and a relative speed of the obstacle viewed from the host vehicle. The obstacle saliency estimating unit estimates the saliency of the obstacle based on the image saliency of the obstacle and a determination result of whether or not the obstacle is located in a driver's blind spot. The external environment recognition device for a vehicle according to claim 1 or 2 . An image acquisition unit for acquiring an image around the host vehicle;
A pixel saliency estimation unit that estimates saliency of each pixel of the image based on a luminance component based on color information of the entire image;
A position information acquisition unit for acquiring position information of obstacles around the host vehicle;
An obstacle saliency estimating unit that estimates the saliency of the obstacle based on the saliency of each pixel at the position of the obstacle in the image;
A vehicle control determination unit that determines whether or not output of an execution command for obstacle avoidance control in the vehicle is based on the estimated saliency of the obstacle, and an output timing;
The pixel saliency estimation unit, the entire image of the luminance component, so as to calculate the image saliency of each pixel of the image using the saliency map based on at least one component of the color components and the edge angle component Configured,
An obstacle detection unit capable of detecting the obstacle by image processing from the image;
The obstacle detection unit detects an obstacle from a low-salience area of the image calculated based on the saliency of each pixel of the image, and outputs the position information of the obstacle to the position information acquisition unit To
Vehicle external recognition device. The obstacle saliency estimation unit calculates the image saliency of the obstacle calculated based on the saliency of each pixel at the position of the obstacle in the image, and whether or not the obstacle is located in a driver's blind spot. The vehicle external environment recognition device according to claim 4 , wherein the saliency of the obstacle is estimated based on the determination result. The vehicle according to claim 4 or 5 , wherein the vehicle control determination unit changes an output timing of an obstacle avoidance control execution command in the vehicle based on the position information of the obstacle and the saliency of the obstacle. External environment recognition device.