JP2011170568A - Device and method for detecting obstacle - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology capable of detecting a distant obstacle on a road and reducing a processing load while suppressing a capacity of a storage area necessary in processing to detect the obstacle. <P>SOLUTION: The technique extracts, from an image photographed from a vehicle, an image area as an area of the road on which the vehicle travels, enlarges a faraway area in the extracted image area to generate an enlargement image, and stores the generated enlargement image. For comparing the enlargement images which are stored sequentially in time, the technology calculates a moving distance on the basis of speed information about the speed of the vehicle, converts the enlargement image which is temporally preceding into a conversion enlargement image corresponding to the moving distance, compares the enlargement image with the converted conversion enlargement image to detect different points as the obstacle in both the images, and outputs the detected detection result. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an obstacle detection apparatus and an obstacle detection method.

  2. Description of the Related Art A device that captures, for example, the front of a vehicle with a camera mounted on the vehicle and detects an obstacle ahead of the vehicle is known. For example, in Patent Document 1, the front of a vehicle is photographed by a camera mounted on the vehicle, a difference is obtained after projective transformation is performed on the captured images at two times before and after, and a feature point in which a time lag has occurred is detected from the difference. Then, an optical flow of the motion vector is detected from these feature points to detect an obstacle ahead of the vehicle.

Japanese Patent No. 3463858

  A technology is known that uses a sensor, a radar, and a GPS to notify a driver of danger through an alarm or monitor when a vehicle approaches an obstacle. Also, in order to make the technology that can be provided at a low price by reducing the number of devices used compared to these technologies, a monocular monochrome camera (hereinafter also simply referred to as a camera) is mounted on a vehicle, and the mounted camera A technique for detecting an obstacle by analyzing an image sequence photographed by the method has been developed.

  In the conventional technique for detecting an obstacle by analyzing an image sequence photographed by a camera, a road plane image (overhead view) is first created. Then, of the created quadrilateral road plane images, the difference between the road plane images at the two preceding and following times is taken, the feature points where the time lag has occurred are detected, and these feature points are detected as obstacles.

  Here, in the above-described conventional technology, it is possible to determine the presence or absence of a distant obstacle by increasing the size of the road plane image, that is, by increasing the length of the road in the front direction of the vehicle. . In other words, in the above-described conventional technology, the larger the road plane image that is created, the greater the distance can be determined.

  However, although the far road obstacle can be detected by enlarging the created road plane image, the larger the road plane image is, the more storage area is required when creating the road plane image. And the processing load of the processing apparatus increases. Note that, by reducing the road plane image, naturally, the capacity of the storage area required when creating the road plane image can be reduced, and the processing load on the processing apparatus can be reduced. However, there is a concern that distant obstacles cannot be detected, and the timing for notifying the driver that an obstacle exists is delayed.

  It is an object of the present invention to provide a technique capable of detecting an obstacle far from the road and reducing the processing load by suppressing the capacity of a storage area required for the process of detecting the obstacle. To do.

In the present invention, in order to solve the above-described problem, the image captured from the vehicle is not converted into a road plane image, but the road far region in the image captured from the vehicle is enlarged and temporally compared with the previous image. It was decided to detect obstacles far away from the road.

  More specifically, the present invention is an obstacle detection device that captures at least one direction around a vehicle and detects an obstacle existing in the one direction from the captured image. An extracting unit that extracts an image region as a region of a road on which the vehicle travels, an enlarged image generating unit that generates an enlarged image by enlarging a distant region in the image region extracted by the extracting unit, and the enlarged image generating unit In order to compare the storage unit that stores the generated enlarged image and the enlarged image that is stored in the storage unit and that moves back and forth in time, the moving distance of the vehicle is calculated based on the speed information related to the vehicle speed, and the time In comparison, the enlarged image conversion unit that converts the previous enlarged image into a converted enlarged image corresponding to the movement distance, and the enlarged image and the converted enlarged image converted by the enlarged image converting unit are different from each other. point Comprising an obstacle detector for detecting as an obstacle, and an output unit for outputting a detection result, wherein the obstacle detecting unit has detected.

  As described above, in the conventional technique for detecting an obstacle using a road plane image, in order to detect a distant obstacle on the road, the road plane image to be created is large, and in an extreme case, to infinity. There was a need to expand. On the other hand, the obstacle detection device according to the present invention can detect an obstacle without creating a road plane image, so that the storage area used is reduced compared to the conventional technique for creating a road plane image. This can reduce the processing load.

  The direction in which the image is taken is not particularly limited as long as it is around the vehicle, but it is preferably forward or backward of the vehicle, particularly forward. The photographing can be performed by photographing means capable of continuously photographing the periphery of the vehicle. A monocular monochrome camera is exemplified as the simplest photographing means, but the photographing means may be a camera corresponding to color. The obstacle in the present invention is an obstacle on the road on which the vehicle travels, and projects upward from the road surface, in particular, moves relative to the road, in other words, an obstacle detection device. Means a vehicle that moves independently of the speed of the vehicle on which it is mounted. Specifically, other vehicles and pedestrians are exemplified as the obstacle.

  The extraction unit extracts an image area including a road area from the image. In addition to the driving lane of the vehicle, the road can include adjacent lanes, roadside belts, and sidewalks. The road area is formed by the boundary between the road specified as described above and another area adjacent thereto. For example, when the road is the traveling lane of the host vehicle, a region sandwiched by the boundary between the traveling lane of the host vehicle and the adjacent lane, roadside zone, and sidewalk corresponds to the road region. For example, when the image is an image in front of the vehicle, the width of the road region is wider in the image as it is closer to the vehicle, and becomes narrower as the distance from the vehicle increases.

  The enlarged image generation unit generates an enlarged image by enlarging a distant area in the extracted image area. The distant area is an area away from the vehicle. For example, when the image is an image in front of the vehicle, the road width is an area that is narrower than a portion close to the vehicle, and an area that is 10 m or more away from the vehicle. Can be the far region. By enlarging the far region, when an obstacle exists in the far region, the obstacle existing in the far region is also enlarged. As a result, it is easy to identify an obstacle, and the accuracy of detecting an obstacle existing in a remote area is improved.

  The storage unit stores the enlarged image. Specifically, the storage unit sequentially stores enlarged images that have been taken and enlarged by the enlarged image generation unit. That is, the storage unit stores a plurality of enlarged images that move back and forth in time. In the present invention, since it is possible to detect an obstacle without creating a road plane image, the capacity of the storage area of the storage unit can be reduced as compared with the conventional case.

The enlarged image conversion unit converts the enlarged image into a converted enlarged image corresponding to the movement distance. Of the enlarged images to be compared, the temporally previous enlarged image, that is, the older enlarged image is converted into the converted enlarged image. When the temporally previous enlarged image is changed in time series in consideration of the movement of the vehicle, it essentially matches the temporally subsequent enlarged image. Therefore, in the present invention, the previous enlarged image in time is converted into a converted enlarged image according to the moving distance. The speed information can be obtained from the vehicle.

  The obstacle detection unit compares the enlarged image with the converted enlarged image converted by the enlarged image conversion unit, and detects a difference between the two images as an obstacle. The converted enlarged image is an image according to the moving distance, and is an image that is changed in time series. This time series change is a change that depends on the speed of the vehicle to the last. In other words, changes that do not depend on the speed of the vehicle are not included. Therefore, by comparing the enlarged image with the converted enlarged image converted by the enlarged image conversion unit, it is possible to extract different points between the two images. The different points appear as changes that do not depend on the speed of the vehicle and correspond to obstacles.

  The output unit outputs the detection result. The output result can be output in a manner that can be recognized by a user of the obstacle detection device, for example, a driver or a passenger of the vehicle. For example, the output result may be output visually through video, or may be output through sound through sound.

  Here, in the present invention, the extraction unit extracts boundary information about a road boundary included in the image, calculates a vanishing point of the image from the boundary information, and uses the vanishing point as a vertex. A triangular image region as a road region may be extracted, and the enlarged image generation unit may enlarge the vicinity of the vanishing point in the image region extracted by the extraction unit to generate a rectangular enlarged image.

  The extraction unit calculates a vanishing point of the image from the boundary information and extracts a triangular image region. Boundary information is information used to distinguish the road on which the vehicle travels from other areas. The lane extends along the road, the boundary provided at the boundary between the roadway and the sidewalk, and provided on the side of the road. Includes information about curbstones and gutters. Here, the vanishing point is an intersection where two straight lines intersect, and is calculated based on the boundary information in the present invention. Therefore, the boundary information includes information for specifying at least two straight lines. The triangular image area has a vertex composed of a vanishing point and two straight lines extending from the vertex. The two straight lines (slanted sides) are lines used for calculating the vanishing point, in other words, lines that distinguish the road on which the vehicle is traveling from other areas adjacent to the road. It should be noted that the position of the base forming the triangular image region specifies to what extent the obstacle is detected in the obstacle detectable region, particularly in the region close to the vehicle.

  The enlarged image generation unit enlarges the vicinity of the vanishing point and generates an enlarged image. Enlarging the vicinity of the vanishing point means that the image in the vicinity of the vanishing point is stretched in the horizontal direction orthogonal to the shooting direction. As a result, the triangular image region is converted into a rectangular enlarged image. The vanishing point corresponds to the position farthest from the vehicle. Accordingly, by enlarging the vicinity of the vanishing point, when an obstacle exists near the vanishing point, the obstacle existing near the vanishing point is also enlarged. As a result, it is easy to identify an obstacle, and the accuracy of obstacle detection is improved.

  In addition to the obstacle detection device described above, the present invention includes an obstacle including at least one of an imaging unit that captures an image of the periphery of the vehicle and a display unit that displays a detection result detected by the obstacle detection. You may comprise as an object detection system.

Moreover, this invention can also be specified as an obstacle detection method which implement | achieves the process performed with the obstacle detection apparatus mentioned above. Specifically, the present invention is an obstacle detection method for photographing an obstacle in the one direction from a photographed image by photographing at least one direction around the vehicle, and from the image, the vehicle An extraction step of extracting an image region including a region of a road on which the vehicle travels, an enlarged image generation step of generating an enlarged image by enlarging a distant region in the image region extracted in the extraction step, and the enlarged image generation step In order to compare the storage step for storing the generated enlarged image and the enlarged image stored in the storage step, the vehicle moving distance is calculated based on the speed information related to the vehicle speed, and the time The enlarged image conversion step for converting the previous enlarged image into a converted enlarged image corresponding to the movement distance, and the converted image by the enlarged image and the enlarged image conversion step. An obstacle that the computer executes an obstacle detection step of comparing the converted enlarged image with each other and detecting a different point between the two images as an obstacle, and an output step of outputting the detection result detected in the obstacle detection step. This is an object detection method.

  In the obstacle detection method according to the present invention, in the extraction step, boundary information regarding a road boundary included in the image is extracted, a vanishing point of the image is calculated from the boundary information, and the vanishing point is calculated. A triangular image area as the road area is extracted as a vertex, and in the enlarged image generation step, the vicinity of the vanishing point in the image area extracted in the extraction step is enlarged to generate a rectangular enlarged image. It may be.

  The present invention may be a program that realizes processing executed by the obstacle detection device described above. Furthermore, the present invention may be a computer-readable recording medium that records such a program. In this case, the function can be provided by causing the computer or the like to read and execute the program of the recording medium. Note that a computer-readable recording medium is a recording medium that accumulates information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read from a computer or the like. Say.

  According to the present invention, it is possible to provide a technique capable of detecting an obstacle far from the road and reducing the processing load by suppressing the capacity of a storage area required for the process of detecting the obstacle. it can.

1 shows a schematic configuration of an obstacle detection apparatus according to an embodiment. Indicates the installation status of the camera. A far obstacle detection processing flow is shown. The display of the straight line by the standard system of Hesse is shown. A state in which a straight line is detected from the left and right of the image is shown. An example of the image image | photographed with the camera is shown. The edge image produced from the image corresponding to FIG. 6A is shown. Corresponding to Figure 6A, showing the sequence _H L, _{H R} representing the presence of a straight line. An image obtained by fitting two straight lines obtained by the Hough transform to the image is shown. The image from which the triangular image area | region was extracted is shown. A rectangular enlarged image in which the vicinity of the vanishing point is enlarged is shown. The conversion enlarged image before in time according to a movement distance is shown. The difference image produced | generated from the difference of an expansion image and the conversion expansion image converted by the expansion image conversion part is shown. This indicates that an edge point that forms one boundary on one side of a broken white line and the boundary on the opposite side of another white line is selected. This indicates that the one that passes through the edge point detected on the same straight line other than the white line and the roadside belt is selected. A method for creating a deformed enlarged image of the next frame (enlarged image later in time) will be described. A camera coordinate system and a vehicle coordinate system are shown. In the projection processing, the pixel value of the intersection (X, Y, Z) of the line of sight that passes from the point (x, y) on the image and passes through the point (x, y) on the image and the road surface X = −h is (x, y) on the image. Shown in the picture. The obstacle detection principle of an object protruding on the road is shown. The principle of detecting obstacles on the road surface is shown. Fig. 4 shows a part of a video image sequence of a total of 750 frames in the embodiment. In the embodiment, a part of an image sequence in which a straight line is detected one by one from the left and right of the image by Hough transform and a black circle is marked at the intersection (vanishing point) is shown. The part of distant area expansion image sequence which expanded the vanishing point vicinity in the right and left in an Example is shown. The obstacle detection result by the process using the overhead view and the obstacle detection result by the distant obstacle detection process are displayed side by side.

  Next, an embodiment of the obstacle detection device of the present invention will be described based on the drawings. Below, it demonstrates as the obstacle detection system 6 containing the camera 2, the obstacle detection apparatus 3, and the display apparatus 4. FIG. The obstacle detection device 3 is mounted on the vehicle 1 and analyzes an image captured by the camera 2 to detect an obstacle 20 on the road in front of the vehicle 1. Hereinafter, in the drawings, reference numeral 21 is given to other vehicles as obstacles, and reference numeral 22 is given to pedestrians as obstacles. In particular, when there is no need to distinguish, it is simply referred to as an obstacle 20. In addition to being displayed on the display device 4, the detection result is output as sound through a speaker (not shown) mounted on the vehicle 1. The embodiment described below is merely an example, and the present invention is not limited to the embodiment described below. For example, the detection result may be output to a storage device of the vehicle 1, an ECU (Electrical Control Unit), or the like, and used for vehicle control such as a PCS (Pre-crash Safety System).

[Constitution]
The obstacle detection system 6 according to the embodiment includes a camera 2, an obstacle detection device 3, and a display device 4. The camera 2, the obstacle detection device 3, and the display device 4 are all mounted on the vehicle 1 and are electrically connected to each other.

  In the present embodiment, as shown in FIG. 2, the camera 2 is provided on the non-mirror surface side of the rearview mirror 11 in order to photograph the front of the vehicle 1. In the present embodiment, a monocular monochrome camera is used as the camera 2 and continuously captures the front of the vehicle 1. The captured image includes information regarding the situation ahead of the vehicle 1 including the obstacle 20. The obstacle 20 in the present embodiment is an obstacle on the road on which the vehicle travels, and examples include other vehicles 21 and pedestrians 22 (including road crossers). The camera 2 may be a camera corresponding to color.

The obstacle detection device 3 analyzes an image taken by the camera 2 and detects an obstacle 20 on the road in front of the vehicle 1. Specifically, the control part 5 is provided and each structure with which the obstacle detection apparatus 3 is provided is controlled. The control unit 5 includes a CPU (Central Processing Unit) 50 and a memory 51.
Etc. and a program executed on the computer. The memory 51 includes a volatile RAM (Random Access Memory) and a nonvolatile ROM (Read Only Memory). ROM includes flash memory, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable
Includes rewritable semiconductor memory such as Read-Only Memory. CPU50 is RO
Various programs for controlling the obstacle detection device 3 stored in M are expanded in the work area of the RAM, and processing according to the various programs is executed according to various data input to the control unit 5. . Each component included in the control unit 5 can be configured as a computer program executed on the CPU 50. Each configuration may be configured as a dedicated processor. Each component included in the control unit 5 includes an extraction unit 52, an enlarged image generation unit 53, a storage unit 54, an enlarged image conversion unit 55, an obstacle detection unit 56, and an output unit 57.

  The extraction unit 52 extracts boundary information about road boundaries 11 and 12 from the image captured by the camera 2, calculates the vanishing point 13 of the image from the boundary information, and uses the vanishing point 13 as a vertex. A triangular image region 16 is extracted as a region (see FIG. 6E).

  The enlarged image generation unit 53 enlarges the vicinity of the vanishing point 13 in the image region 16 extracted by the extraction unit 52 to generate a rectangular enlarged image (see FIG. 6F).

  The storage unit 54 sequentially stores, in a predetermined storage area in the memory 51, enlarged images that have been captured and enlarged by the enlarged image generation unit 53. The memory 51 stores a plurality of enlarged images that move back and forth in time.

  The enlarged image conversion unit 55 converts the previous enlarged image into a converted enlarged image corresponding to the moving distance. The moving distance is calculated based on the speed information. The speed information can be appropriately acquired from within the vehicle 1.

  The obstacle detection unit 56 compares the enlarged image with the converted enlarged image converted by the enlarged image conversion unit 53, and detects a difference between the images as an obstacle. In the present embodiment, the obstacle detection unit 56 generates a difference image between the enlarged image and the converted enlarged image converted by the enlarged image conversion unit 53, and detects the obstacle by performing binarization with an appropriate threshold value. (See FIG. 6H).

[Processing flow]
Next, a process executed by the obstacle detection device 3 according to the above-described embodiment (hereinafter also referred to as a remote obstacle detection process) will be described. Here, FIG. 3 shows a distant obstacle detection processing flow for detecting an obstacle far from the road. The order of the remote obstacle detection process described below is merely an example, and may be appropriately changed depending on the embodiment. In the following description, the front of the vehicle 1 is photographed by the camera 2 mounted on the vehicle 1 traveling at a predetermined speed (see FIG. 2), and the obstacle 20 in front, particularly the obstacle 20 that appears far away is detected. This will be described as an example. Here, in the distant obstacle detection process, a road video image (corresponding to an image of the present invention, hereinafter simply referred to as an image) taken from the camera 2 mounted on the vehicle 1 is not projected onto the road surface, but a road. The far portion is stretched to the left and right, and the rear frame (corresponding to the converted enlarged image corresponding to the temporally subsequent image of the present invention) and the previous frame (converted expanded image corresponding to the temporally previous image of the present invention). The difference image is taken to generate a difference image, and based on this, an obstacle far from the road is detected. Therefore, the distant obstacle detection process is particularly suitable for detecting the obstacle 20 far from the road. On the other hand, the obstacle 20 existing in the area close to the vehicle 1 can be detected by an existing technique for projecting an image onto the road surface (hereinafter also referred to as processing using a bird's-eye view), and is a distant obstacle. The detection process and the process using the overhead view can be used in combination. Therefore, processing using the overhead view will also be described after the description of the remote obstacle detection processing.

(Extract triangle image area)
In step S01, the extraction unit 52 extracts the triangular image region 16 from the image captured by the camera 2. In the present embodiment, the boundary of the white line or the roadside zone is extracted as boundary information regarding the road boundaries 11 and 12 (hereinafter, simply referred to as the road boundary 11 when it is not necessary to distinguish the road boundary). The vanishing point 13 is calculated from the intersection of the two straight lines 14 and 15 calculated by the road boundary 11 (hereinafter referred to as the straight line 14 when there is no need to distinguish the straight line), and the vanishing point 13 is defined as the vertex. A triangular image area 16 as a road area is extracted. Here, FIG. 6A is an example of an image photographed by the camera 2, and FIG. 6B is an edge image created from an image corresponding to FIG. 6A. Figure 6C corresponds to Figure 6A, showing the sequence _H L, _{H R} representing the presence of a straight line. FIG. 6D shows an image obtained by fitting two straight lines obtained by the Hough transform to the image. Furthermore, FIG. 6E shows an image from which a triangular image region has been extracted.

(Line voting and detection in Hough transform)
Here, the extraction of the triangular image region 16 by the extraction unit 52 can be performed by Hough transform. First, the Hough transform will be described. The Hough transform is a method for determining a straight line that passes through the most feature points of an image. In other words, the Hough transform is to determine a straight line that best fits an image and is a technique widely used in image processing. When this method is used, the straight line 14 is represented by the standard type of Hesse shown in Equation 1. FIG. 4 shows a straight line display according to the Hesse standard system. FIG. 5 shows a state in which straight lines are detected from the left and right sides of the image.

Equation 1 represents a straight line 14 whose distance from the origin is h and whose unit normal vector n forms an angle θ from the x axis. That is, according to the Hesse standard system, the straight line 14 can be uniquely represented by the distance h from the origin O and the angle θ. In this embodiment, the center of the image is the coordinate origin O, the x axis is upward, and the y axis is right. A road boundary 11 consisting of a roadside band or a white line is detected from the left and right of the image. The range of the angle between the x axis and the straight line is 30 °, the range of the distance is 100 pixels, and the direction of the straight line on the left side θ _L The distance h _L and the range of the straight line direction θ _R on the right side and the distance h _R are set as shown in Equation 2, for example. The setting range is not limited to the following, and can be set based on past data and experimental data relating to the shape of the straight line 14 formed of a white line or a roadside band reflected in the image.

Once the setting range is set, 31 × 101 integer arrays H _L (i, j) and H _R (i, j) corresponding to the left and right straight line 14 and 15 candidates are prepared. Then, the edge pixel (x, y) is extracted from the left half and the right half of the edge image (see FIG. 6B), and the following calculation is performed for i = 0, 1,.

First, the left half and the right half of the image, the orientation theta _R orientation theta _L and the right of the straight line 14 on the left side of the straight line 15 respectively determined (see Equation 3).

Next, for the left half and the right half of the image, a distance h _L from the origin O to the left straight line 15 and a distance h _R from the origin O to the right straight line 14 are obtained (see Equation 4).

Then, the left half and the right half of the image, and updates the sequence H _L, the value of H _R as number 5. It should be noted that anything outside the 31 × 101 range of the array is ignored.

The values of the elements (i, j) of the arrays H _L and H _R obtained in this way are the values of the straight lines {h _R , θ _R }, {h _L , θ _L } corresponding to the (i, j). The number of neighboring weighted pixels. Therefore, assuming that the maximum elements of H _L and H _R are H _L (i _L , j _L ) and H _R (i _R , j _R ), respectively, the straight lines 14 and 15 represented by the equations are as shown in Equation 6, respectively.

Incidentally, FIG. 6C corresponds to Figure 6A, showing the sequence _H L, _{H R} representing the presence of a straight line. FIG. 6D shows an image obtained by fitting the two straight lines 14 and 15 obtained by the Hough transform to the image.

(Calculation of vanishing point)
Next, calculation of the vanishing point 13 will be described. The vanishing point 13 is calculated as an intersection of two straight lines 14 and 15 expressed by Equation 6. Specifically, the coordinates of the vanishing point 13 are obtained as follows. First, vectors n _L and n _R are defined using θ _L , h _L , θ _R and h _R obtained from Equation 6 (see Equation 7).

Next, a vector product u of the two vectors n _L and n _R is obtained (see Expression 8).

As a result, the intersection (x _v , y _v ) of the two straight lines 14 and 15 is obtained (see Equation 9).

  Here, as described above, the Hough transform is to detect a straight line passing through as many edge points as possible on the same straight line, but even if the edge points are arranged on a straight line, it is the boundary of the same straight line. It is not always a line. For example, since a scene far away from a road has a small range in the original image, it is not always possible to accurately detect an edge point when detecting an edge. Therefore, it is also assumed that the straight line 14a connecting one boundary of the broken white line and the boundary on the opposite side of the other white line applies. FIG. 7 shows that an edge point is selected which forms one boundary on one side of a broken white line and the boundary on the opposite side of another white line. In addition, the straight line 15a connecting the boundary between the building 7 and the sky and an area where a large number of edge points on the road are detected (such as an area where characters such as “Torare” and “Bus only” are written) is detected. It also exists. FIG. 8 shows that the one that passes through the edge point detected on the same straight line, which is different from the white line and the roadside belt, is selected.

  In order to solve the above-described problems, a “direction” may be attached to each edge point. Specifically, the following procedure is followed. Since edge detection uses differentiation of the luminance value of the image, the direction in which the volatility value I increases most is calculated at each edge point (see Equation 10).

However, I _x and I _y are differentiations of the luminance value I in the x-axis direction and the y-axis direction, respectively. Therefore, in Equation 10, when the direction angle of ▽ I is φ, cos φ and sin φ are as shown in Equation 11.

  Here, the inner product of the unit vector in this direction (see Equation 12) and the unit normal vector of the straight line 14 (see Equation 13) is as shown in Equation 14.

The inner product c calculated by Equation 14 is positive when the luminance value increases in the direction of the straight normal vector n, and is 0 when the luminance value increases in the direction of the straight line. Negative if it increases in the opposite direction. Therefore, by voting a real number c instead of an integer to the matrix H of the Hough transform, for example, in the straight line 14 that is applied so as to cross the white line 14, the sign of c of the edge point of a different part is reversed and the vote is voted. The value is closer to 0, making it difficult to select. Specifically, the above formula 5 is updated to formula 15.

Then, the elements having the maximum absolute values of H _L and H _R are set as H _L (i _L , j _L ) and H _R (i _R , j _R ), respectively, and the above calculation is repeated. As a result, it becomes possible to more accurately detect the road boundaries 11 and 12 including white lines and roadside belts.

After detecting the boundary 11 of the road, then the x-coordinate of the boundary between the road surface of the left and right edges of the image and x _a. The road surface may be a surface that can be regarded as a road surface. As a result, the road region in the image has two points (x _a1 + (M−1) / 2) with the vanishing point 16 (x _v , y _v ) estimated from the white line detection by the Hough transform as the vertex as described above. , (X _a1 − (M−1) / 2) is a triangular image area 16 having a horizontal line as a base. M is the width of the image.

  Thus, the extraction of the triangular image region 16 in step 01 is completed. When the extraction of the triangular image region 16 is completed, the process proceeds to step S02.

(Generate enlarged image)
In step S02, the magnified image generating unit 53 magnifies the vicinity of the vanishing point 13 in the triangular image region 16 extracted by the extracting unit 52 to generate a quadrangle magnified image (see FIG. 6F). Specifically, the following procedure is performed. As described above, the triangular image region 16 in the image has two vanishing points (x _v , y _v ) estimated from the straight line 14 generated from the road boundary 11 by the Hough transform as described above. This is a triangular image region 16 having a horizontal line connecting (x _a1 + (M−1) / 2) and (x _a1 − (M−1) / 2) as a base. Since the distant area on the road is compressed near the vertex (x _v , y _v ) of this triangle, that is, near the vanishing point 13, the accuracy is not sufficient as it is, and analysis is difficult. Therefore, the vanishing point 13 (x _v , y _v ) which is one of the vertices of the triangular image region 16 is stretched left and right to be converted into a rectangular region (see Expression 16).

  In Equation 16, when solving for x and y, respectively, Equation 17 is obtained.

Then, a storage area for storing an image obtained by enlarging the triangular image area 16 is prepared in advance in the memory 51, and each point (x ′, y ′) of the image area 16 is expressed by Equation 17 ( x, y) is calculated and the point on the image is copied to (x ′, y ′). Since x = x _v (horizontal line passing the vanishing point) is a singular point, it is better not to calculate. Thus, occurrence of division by zero can be suppressed in Equation 16. That is, the processing load can be reduced.

  Thus, the generation of the enlarged image in step 02 is completed. When the generation of the enlarged image is completed, the process proceeds to step S03.

(Conversion to converted enlarged image)
In step S03, the enlarged image conversion unit 55 converts the temporally previous enlarged image into a converted enlarged image corresponding to the movement distance (see FIG. G). Here, as described above, even if the vicinity of the vanishing point 13 is enlarged, the position of the straight line y ′ = 0 is at the same position on the image. Therefore, when the point on the straight line is mapped onto the road surface, the point on the Z-axis as shown in Equation 18 is obtained.

Also, assuming that the vehicle 1 is moving at a speed V (pixel / frame), the position on the Z-axis of the straight line y ′ = 0 one frame before (temporally before) is as shown in Equation 19. .

  The x ′ coordinate of the position obtained by inversely mapping this point on the image is as shown in Expression 20.

Therefore, in order to match the deformed enlarged image with the deformed enlarged image one hour before in time, it is necessary to perform processing as shown in FIG. FIG. 9 shows a method of creating a deformed enlarged image of the next frame (enlarged image after time). Specifically, first, Equation 18 is calculated for each point (x ′, y ′) of the deformed enlarged image, and a point on the straight line is mapped onto the road surface to obtain a point on the Z axis. Next, assuming that the vehicle is moving at a speed V (pixel / frame), the position on the Z-axis of the previous deformation enlarged image in terms of time is obtained from Equation 19. Next, the x ′ coordinate of the position obtained by inversely mapping the position on the Z axis on the image is obtained from Equation 20. Then, the luminance value of the point (x ′ ₋₁ , y ′) of the previous deformed enlarged image in time is copied to the point (x ′, y ′).

  Thus, the conversion into the converted enlarged image is completed. When the conversion to the converted enlarged image is completed, the process proceeds to step S04.

(Detection of obstacles)
In step S04, a difference image between the enlarged image and the converted enlarged image converted by the enlarged image conversion unit 55 is generated, and the obstacle 20 is detected by performing binarization with an appropriate threshold (FIG. 6).
H). Thereafter, the output unit 57 outputs the detection result to the display device 4. When outputting to the display device 4, it is preferable to make it easier to identify by blinking around the detected obstacle or coloring the obstacle. In addition, while displaying on the display apparatus 4, you may make it alert | report that an obstruction exists with a sound through a speaker.

(Processing using an overhead view)
The far obstacle detection process described above can be used in combination with the process using the overhead view as described above. Therefore, processing using an overhead view for projecting an image photographed by the camera 2 onto a road surface will be described below. In the case where the distant obstacle detection process and the process using the overhead view are used in combination, for example, for the detection of the obstacle 20 that is 10 m or more away from the vehicle 1, the distant obstacle detection process is used. For the detection of the obstacle 20 existing in a closer area, a process using an overhead view can be used. The processing described below can be performed by the CPU 50 of the control unit 5.

  Here, FIG. 10 shows the camera coordinate system and the vehicle coordinate system in the processing using the overhead view, and FIG. 11 shows the point (x, y) on the image that is out of the viewpoint O in the processing using the overhead view. A state in which the pixel value of the intersection (X, Y, Z) between the line of sight passing through and the road surface X = −h appears in (x, y) on the image.

  In the processing using the overhead view, a road plane image (overhead view) as if the image taken by the camera 2 was taken from above the road is created, and the current road plane image and the previous road plane image in time are created. An obstacle is detected from the difference of the road plane image of the previous frame.

  First, the vehicle image system and the camera image system will be described. The vehicle image system has the viewpoint (lens center) of the camera 2 as the origin, the X axis is the vertical upper direction, the Y axis is the horizontal horizontal direction, and the Z axis is the horizontal front. The camera coordinate system takes the Z axis in the optical axis direction of the camera (diagonally below), the Y axis in the horizontal and horizontal directions (matches the Y axis of the vehicle image system), and the X axis so as to be orthogonal to the Y axis. It is assumed to be obliquely upward. At this time, if the camera 2 is inclined downward by an angle α, the vehicle image system (left side of Equation 21) and the camera image system (right side of Equation 21) are expressed by Equation 21.

  When the road surface is -h with respect to the vehicle image system, a point (X, Y, Z) on the road surface that appears as a point (x, y) on the image surface is expressed by Equation 22. h is the vertical height from the road surface to the camera 2.

  Further, when the point (X, Y, Z) on the road surface is converted into the camera image system, the following equation 23 is obtained.

  Next, creation of a road plane image will be described. First, the point (x, y) on the road surface in the camera image system is mapped to the point on the road surface in the vehicle image system using the relationship of Equation 23. At this time, the coordinate system divided by the height h of the camera 2 is represented by Equation 24, and the coordinate system represented by Equation 24 is considered as the coordinate system on the road surface.

Here, the vertical (x direction) width of the image is N, and the horizontal (y direction) width of the image is M, and the boundary between the road surface or the horizontal ground and a vertical object (such as a building) at the left and right edges of the image. When the position of the surface is (x _a1 ± (M−1) / 2), the point (x, y) on the road surface in the camera image system is mapped to the point shown in Equation 25.

Further, if a boundary where the bonnet portion of the vehicle at the left and right edges on the image is reflected is (x _b1 ± (M−1) / 2), the point (x, y) on the road surface in the camera image system is It is mapped to the point shown in FIG.

  Here, a rectangular area represented by Equation 27 on the road surface is considered.

  The maximum value of the point Z is determined by how far forward is displayed. Therefore, when all the fronts are displayed, the maximum value of the point Z is infinite. If the rectangular area shown in Equation 27 is square, the maximum value of the point Z is as shown in Equation 28.

  Here, in Expression 27, the larger the maximum value of the point Z, the larger the planar image to be created, and it is possible to determine whether there is an obstacle farther away. However, at the same time, the capacity of the storage area of the storage device required for creating the road plane image increases, and the processing load on the processing device increases. Furthermore, as the distance from the vehicle to the obstacle increases, the range that appears on the captured image is naturally reduced, and since the area is enlarged and written, a decrease in determination accuracy is unavoidable. Note that by reducing the maximum value of the point Z, it is possible to reduce the capacity of the storage device required when creating a road plane image and reduce the processing load on the processing device. However, there is a concern that the timing to inform the driver that there is an obstacle will be delayed. Therefore, based on the above, it is necessary to determine the rectangular area represented by Equation 7, in other words, the minimum value and the maximum value of the display area. Therefore, in the present embodiment, the far obstacle detection process described above is used for the detection of the obstacle 20 in the distance.

  When the rectangular area represented by Equation 27, in other words, the minimum value and the maximum value of the display area are determined, the storage area (image buffer) of the storage device corresponding to the display area is determined, and the image is mapped. Specifically, a pixel (x, y) on the image plane corresponding to each pixel of the display area for which the range is determined is calculated, and the luminance value of the pixel is expressed in the coordinate system represented by Equations 24, 25, and 26. Copy to each point. By performing this operation for all the pixels in the display area, the inverse mapping shown in Equation 29 is obtained.

  Here, since the calculated inverse mapping (x, y) is not necessarily an integer value, it is determined by bilinear interpolation of surrounding pixels.

  Next, bilinear interpolation will be described. Bilinear interpolation is first performed to obtain a luminance value at a position of a real number (i, j) from an image I of size M × N, and is performed as follows. Hereinafter, k, l, ε, and η are real numbers, and K and L are integers.

  First, in the case of k <0, or k ≧ M, L <0, or l ≧ N, processing is performed assuming that the region protruding from the frame is 0. Thereby, (k, l) can take an infinite range.

  Next, when k <0, or k ≧ M, L <0, or l ≧ N, the integer parts K and L and the fractional parts ε and η are calculated as in Expression 30. Note that the right side of the leftmost expression in Equation 30 is a floor function (floor) representing the integer part (cut off) of k.

  Next, I (I, J) and I (I, J + 1) are internally divided into η: 1−η, and I (I + 1, J) and I (I, J + 1) are internally divided into η: 1−η. , They are internally divided into ε: 1−ε (see Equation 31).

  Next, detection of the obstacle 20 will be described. First, the image ahead of the vehicle 1 is mapped onto the road surface as described above. The road surface is assumed to be horizontal, and the vehicle 1 travels straight at a speed of V / frame. At that time, it can be interpreted that the object on the road is approaching the vehicle at a relative speed of −V / frame. That is, a point on the road surface can be interpreted as being translated vertically downward by V / h between frames. Therefore, if the road plane image of the previous frame is translated vertically downward by V / h, it matches the current frame. However, if there is an object that protrudes on the road surface, that is, an obstacle 20 such as a pedestrian 22 or a vehicle 21, the image is different from the road surface and does not translate. Specifically, the further away from the road surface, that is, the higher the height, the lower the vehicle moves at a higher speed than the road surface. Thus, the non-horizontal part on the road surface moves differently from the road surface. Therefore, it is possible to detect a non-horizontal portion on the road surface by calculating a difference image between the previous frame and the current frame from which the position shift due to the time difference is removed by translation and binarizing with an appropriate threshold. Become.

Specifically, the obstacle image _Et is created by the following procedure. First, a road plane image I ₀ ′ is created using the 0th image I ₀ as an image. Next, the first image I ₁ is used as an image, and the same surface image I ₁ ′ is created. Next, the road plane image I ₀ ′ is translated vertically downward, overlapped with I ₁ ′, a difference is taken, and a binarized difference image D1 is created. Then, the second image is used as an image, and the above processing is repeated thereafter.

  Here, FIG. 12 shows the obstacle detection principle of an object protruding on the road. The line segment AB perpendicular to the road surface moves to A′B ′ relative to the vehicle at the next time. At this time, the map A ′ of the upper A on the road surface moves to A ′ at a speed higher than the moving speed V of the vehicle.

  FIG. 13 shows the principle of detection of obstacles on the road surface. An object having a height has a larger moving distance on the road plane as it gets away from the road surface, and a difference is generated when a difference between frames is taken. On the other hand, since a road sign or the like existing on the road surface has no height, there is no difference even if an interframe difference is taken.

  As described above, it is possible to detect an obstacle, in particular, an obstacle that is close to the vehicle 1. However, in the process using the overhead view, the obstacle detection status is displayed as an overhead view. Therefore, in order to make it easier for the driver to check the obstacle, the obstacle image Et is mapped to an image taken in front. Specifically, a storage area corresponding to the display area of the image frame is prepared in the memory 52, and the position of the road plane image corresponding to each pixel (x, y) is detected as an obstacle. For example, if the maximum luminance value is reached, the pixel (x, y) in the buffer area is displayed in color. Specifically, the following procedure is used.

  At each time t, each pixel (x, y) of the original image It ′ is manipulated to calculate the position on the road surface shown in Equation 32.

  If the pixel of the obstacle image Et corresponding to the position shown in Expression 32 is a black pixel or is outside the frame, nothing is done. On the other hand, if the pixel of the obstacle image Et corresponding to the position shown in Expression 12 is a white pixel, the R value of each pixel (x, y) of the original image It ′ is set to the maximum gradation, and the G value and the B value are set. Rewrite to 0.

(Function and effect)
According to the obstacle detection device according to the embodiment described above, in the detection of an obstacle far from the road, it is possible to detect the obstacle without creating a road plane image. The storage area used can be reduced compared with the case where an obstacle far from the road is detected by processing, and the processing load can be reduced.

(Example)
How far can the obstacle 20 be detected by enlarging the vicinity of the vanishing point 13 estimated from the detection of the straight lines 14 and 15 by the Hough transform and the straight lines 14 and 15 fitted by the Hough transform and taking the difference image? We experimented. The setting parameters of the camera 2 are as shown in Table 1. In addition, the determination area for determining obstacles far from the road is as shown in Equation 33. In Equation 33, _xv is the x coordinate of the vanishing point 13.

  From the above experiment, it was confirmed that the straight lines 14 and 15 can be automatically and accurately applied to the boundary between the white line and the road in the image by using the Hough transform. The point where the two straight lines 14 and 15 fitted to each other is the vanishing point 13, and a distant obstacle 20 was detected by enlarging the vicinity. The image was a video image sequence of a total of 750 frames (see FIG. 14). 14 to 17 exemplify 40 frames, 180 frames, and 430 frames among all 750 frames. The video image sequence goes straight on a two-lane straight road with good visibility, and pedestrians 22 frequently cross the front of the vehicle 1. In addition, in order to avoid vehicles parked on the road, it is considered dangerous for the vehicle 1 such that the oncoming vehicle runs out of the center line or another vehicle 21 departs from the garage in front of the vehicle 1. This is a setting where many factors are present. This input video image sequence was analyzed for obstacles far away from the road, and the difference from the processing using the overhead view was confirmed. FIG. 15 shows an image in which straight lines are detected one by one from the left and right of the image by the Hough transform, and the intersection (disappearance point 13) is marked with a black circle.

Further, a far region enlarged image in which the vicinity of the vanishing point 13 is enlarged to the left and right is shown in FIG. Finally,
FIG. 17 shows the obstacle detection result by the process using the overhead view and the obstacle detection result by the far obstacle detection process displayed side by side. A region surrounded by dots in the image is a region where an obstacle is determined. As can be seen from FIG. 17, in the processing using the overhead view, the determination area of the obstacle 20 is wide immediately before the vehicle 1, but the detection range is at best considering the capacity of the storage area and the processing load. I could only take about 10 meters in front of the vehicle. In addition, while the pedestrian 22 located at a distance of about 10 meters has a slight obstacle determination at his / her feet, in this distance obstacle detection process, the distance from the vehicle 1 is some distance away. It is possible to perform obstacle determination in the area where In FIG. 17, a pedestrian 22 that can be slightly detected as an obstacle 20 in the process using the overhead view can also detect an obstacle so as to surround the whole body of the pedestrian 22 by the distant obstacle detection process. It was. Moreover, the pedestrian 22 of FIG. 17 which was outside the detection range in the process using the overhead view could be determined as the obstacle 20 (see 400 frame of FIG. 17).

  The preferred embodiments of the present invention have been described above, but the obstacle detection device according to the present invention is not limited to these, and can include combinations thereof as much as possible.

DESCRIPTION OF SYMBOLS 1 ... Vehicle 2 ... Camera 3 ... Obstacle detection apparatus 4 ... Display apparatus 5 ... Control part 6 ... Obstacle detection system 11, 12 ... Boundary 13 ... Disappearance Points 14, 15 ... straight line 16 ... triangular image area 20 ... obstacle 21 ... other vehicle 22 ... pedestrian 50 ... CPU
51 ... Memory 52 ... Extraction unit 53 ... Enlarged image generation unit 54 ... Storage unit 55 ... Enlarged image conversion unit 56 ... Obstacle detection unit 57 ... Output unit

Claims (4)

An obstacle detection device that captures at least one direction around a vehicle and detects an obstacle existing in the one direction from the captured image,
An extraction unit for extracting an image region including a region of a road on which the vehicle travels from the image;
An enlarged image generation unit that generates an enlarged image by enlarging a distant region in the image region extracted by the extraction unit;
A storage unit for storing an enlarged image generated by the enlarged image generation unit;
In order to compare the enlarged images that are stored in the storage unit and that move forward and backward in time, the moving distance of the vehicle is calculated based on the speed information related to the speed of the vehicle, and the previous enlarged image in time is determined according to the moving distance. An enlarged image conversion unit for converting to a converted enlarged image;
An obstacle detection unit that compares the enlarged image and the converted enlarged image converted by the enlarged image conversion unit to detect different points of both images as an obstacle,
An output unit for outputting a detection result detected by the obstacle detection unit;
An obstacle detection device comprising: The extraction unit extracts boundary information about a road boundary included in the image, calculates a vanishing point of the image from the boundary information, and has a triangle as a road region having the vanishing point as a vertex. Extract the image area,
The enlarged image generating unit generates a rectangular enlarged image by enlarging the vicinity of the vanishing point in the image region extracted by the extracting unit,
The obstacle detection apparatus according to claim 1. An obstacle detection method for photographing at least one direction around a vehicle and detecting an obstacle existing in the one direction from a photographed image,
An extraction step for extracting an image region including a region of a road on which the vehicle travels from the image;
An enlarged image generation step of generating an enlarged image by enlarging a distant area in the image area extracted in the extraction step;
A storage step of storing the enlarged image generated in the enlarged image generating step;
In order to compare the enlarged images that are stored in the storage step and that move forward and backward in time, the moving distance of the vehicle is calculated based on the speed information related to the speed of the vehicle, and the previous enlarged image in time is determined according to the moving distance. An enlarged image conversion step for converting to a converted enlarged image;
An obstacle detection step of comparing the enlarged image and the converted enlarged image converted by the enlarged image conversion step to detect different points of both images as obstacles;
An obstacle detection method in which a computer executes an output step of outputting a detection result detected in the obstacle detection step. In the extraction step, boundary information relating to a road boundary included in the image is extracted, a vanishing point of the image is calculated from the boundary information, and a triangle as a road region having the vanishing point as a vertex is calculated. Extract the image area,
In the magnified image generating step, a quadrangle magnified image is generated by enlarging the vicinity of the vanishing point in the image region extracted in the extracting step.
The obstacle detection method according to claim 3.