JP2011076214A - Obstacle detection device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent a change of an object such as a road surface from being falsely determined while allowing various objects to be determined as an obstacle. <P>SOLUTION: The obstacle detection device divides an input image (color still image) of a road surface R captured by a camera into a plurality of regions based on a frame F, and extracts average luminance or average color saturation as an input factor for each divided region. The obstacle detection device calculates the distance in a reference space based on the input factor, and determines that there is an obstacle when the distance exceeds a threshold. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an obstacle detection apparatus that determines an obstacle within a target range using an image photographed by a camera.

  With respect to this type of obstacle detection device, there is conventionally known a prior art for detecting a stationary obstacle with a single camera instead of a stereo type (see, for example, Patent Document 1). In this prior art, for example, when the direction in which the vehicle is going to move backward (back) is photographed with an in-vehicle camera, the road (road surface) portion is generally a grayscale image having no color, but the other portions are Since the image is a chromatic image having color information, attention is focused on the fact that the road surface portion and other portions can be distinguished based on the color information in the image. For this reason, according to the prior art, it is considered that the presence of a stationary obstacle can be detected from even a still image taken by a single camera.

JP 2009-126493 A

  However, since the above prior art performs detection (determination of presence / absence) of an obstacle depending only on color information in the image, for example, if the object reflected in the image is a person dressed in red or the like, Although this is recognized as an obstacle, a person whose entire body is unified with achromatic clothing such as black or white cannot be recognized as an obstacle.

  In other words, since the prior art method determines an obstacle depending on only one factor of color information in the image, if the factor is significantly different from the road surface portion, this can be easily determined. Can be determined. However, on the other hand, an object for which the only factor of color information is hardly different from the road surface portion cannot be determined as an obstacle even if other factors are greatly different. As described above, the prior art methods have a problem that there are few variations of objects that can be recognized as obstacles, and the recognition performance of the obstacles is not yet sufficient.

  In this regard, although a method of determining an obstacle by observing a plurality of factors (for example, color information and brightness information) in a still image is conceivable, in this case, a change in road surface condition at the time of shooting (for example, sunshine) The result of the determination is likely to be sensitive to a change in brightness such as lighting or lighting), and an object that is not an obstacle may be erroneously determined as an obstacle.

  Therefore, an object of the present invention is to provide a technique that makes it difficult to make an erroneous determination with respect to a change in an object such as a road surface while ensuring diversity of objects that can be determined as an obstacle.

In order to solve the above problems, the following solution means of the present invention are employed.
The obstacle detection apparatus of the present invention includes a camera that captures a predetermined target range, and a control unit that determines the presence or absence of an obstacle within the target range using an input image obtained by capturing with the camera. In the present invention, the control means determines obstacles by the following method. Note that the following processes (1) to (3) are merely indexes provided for convenience of disclosure of the present invention. Here, the determination (detection) method of the present invention is the only procedure or step. It is not intended to be limited to.

(1) Extracting a plurality of types of input factors The control means extracts (calculates) a plurality of types of input factors for each of a plurality of divided regions obtained by dividing the input image captured by the camera. The “input factor” extracted here can be derived, for example, by a data calculation process using an input image taken by a camera. Further, by dividing the input image into a plurality of regions, it is possible to determine obstacles with higher accuracy for the target range.

(2) Distance calculation for each divided region Next, the control means calculates a distance for each divided region of the input image based on the extracted plural types of input factors. The “distance” calculated here is a statistical index obtained when each position is applied to each divided area in a reference space formed by a plurality of types of reference factors stored in advance for obstacle determination. It is a value that becomes a scale or the like.

  A plurality of types of reference factors forming the above “reference space” are obtained by extracting a plurality of reference images prepared separately from the input image for each divided region obtained by dividing each of the reference images into a plurality of regions. Since the control means forms the above-mentioned “reference space” when calculating the “distance”, these plural types of reference factors can be stored in advance.

  In addition, the “reference image” can be obtained by, for example, photographing in advance a standard object (for example, a general road surface in a state where no obstacle exists) with respect to a target range photographed by a camera. Since a plurality of “reference images” are used at this time, for example, even when the target is a road surface (including the ground), various changes (differences in road surface conditions, environment, brightness, etc.) are assumed in advance. Multiple types of reference factors can be extracted. By extracting a plurality of types of reference factors for each divided region of such a “reference image”, the statistical distance distribution in the reference space for a standard object (reference image) can be made known from there. it can. Therefore, the “distance” calculated based on a plurality of types of input factors newly in the reference space serves as a scale for determining the position of each divided area in the reference space in the input image.

(3) Obstacle determination The control means determines the presence or absence of an obstacle in the target range for each divided region of the input image based on the calculated distance. The “distance” calculated as described above indicates where each divided region of the current input image exists in the reference space. For this reason, for example, when the calculated “distance” exceeds a preset threshold value, it is determined that “there is an obstacle”, and otherwise, it can be determined that “there is no obstacle”.

  In other words, if there is a divided area where the “distance” exceeds the threshold, the “distance” of the divided area is considered to be far from the known distribution of the “reference image”. ) Can be determined that some kind of obstacle is reflected (there is an obstacle). On the other hand, if the “distance” does not exceed the threshold for any of the divided areas, it is considered to be close to the known distribution of the “reference image”, and therefore an obstacle appears in the input image (in the divided area). It can be judged that there is no (no obstacle).

  As described above, according to the obstacle detection device of the present invention, since the calculation of the “distance” used for the obstacle determination is performed using a plurality of types of input factors, it is possible to determine the obstacle even for a wider variety of objects. , It can increase the variation of obstacles that can be recognized. In addition, since the reference factors extracted using multiple types of reference images are stored in advance, the presence or absence of obstacles can be accurately determined without excessively reacting to various changes in the target range captured by the camera. Judgment can be made.

  More preferably, the control means determines the obstacle using the Mahalanobis distance. That is, the control means calculates the Mahalanobis distance for each divided area of the input image in the Mahalanobis space as the distance in the reference space, with the Mahalanobis space formed by the plurality of types of reference factors in (2) above as the reference space. In (3) above, the presence / absence of an obstacle in the target range is determined for each divided region of the input image based on the calculated Mahalanobis distance.

  In this case, the distribution of the Mahalanobis distance obtained for each divided area with respect to the reference image is known, and the presence of an obstacle is easily determined by calculating and applying the Mahalanobis distance obtained for each divided area from the input image. can do. That is, the larger the Mahalanobis distance calculated for a certain divided area, the farther the divided area is from the distribution of the reference image in the Mahalanobis space. For this reason, for example, a threshold is set in advance for the Mahalanobis distance that can be determined that there is no obstacle, and if this threshold is exceeded, it is determined that there is an obstacle, and otherwise there is no obstacle. it can.

  More practically, it is assumed that the control means determines the road surface type from the input image when determining the obstacle. That is, the control unit divides the plurality of reference images prepared separately from the input image in the above (2) into a plurality of regions for each of a plurality of road surface types included in the target range photographed by the camera. A plurality of types of reference factors obtained by extraction for each divided region are stored in advance for each of a plurality of road surface types. The control means calculates a distance for each divided region of the input image in the reference space for each of a plurality of road surface types based on a plurality of types of input factors, and sets a road surface type included in the target range based on the calculated distance. . In the above (3), the control means determines the presence or absence of an obstacle in the target range for each divided region based on the distance calculated for the set road surface type.

  In such a mode, even if the target range to be photographed by the camera includes an unknown road surface type, any one of them is calculated based on the distance calculated for each of the plurality of road surface types (may be Mahalanobis distance). The road surface type can be set. That is, the control means stores a plurality of types of reference factors for each of a plurality of road surface types (for example, road surface types a to d), so that a reference image for each of the plurality of road surface types a to d prepared in advance is stored. Any of the distance distributions for can be known. Then, when distances (for example, distances Da to Dd) are calculated for each of a plurality of road surface types a to d based on a plurality of types of input factors extracted from the current input image, the distance is the shortest among them. Any road surface type (for example, road surface type b) can be set (estimated) as the road surface type included in the current target range.

  In addition, when the distance calculated for the set road surface type exceeds, for example, a threshold value, the control unit can determine that there is an obstacle in the divided region, and can determine that there is no obstacle in other areas.

  As described above, the obstacle detection device of the present invention can ensure a large number of variations of objects that can be recognized as obstacles, and enables more accurate determination.

  Even if the target range to be photographed by the camera includes an unknown road surface type, it is possible to set the road surface type using the input image and determine the presence or absence of an obstacle with high accuracy. For this reason, for example, a vehicle mounted on a moving means that moves on various road surfaces (including the ground) such as a vehicle (automobile, truck / bus, two-wheeled vehicle, etc.) is used to remove obstacles on the road surface. Suitable for detecting applications.

It is a schematic diagram which shows embodiment at the time of mounting an obstacle detection apparatus in a vehicle. It is the functional block diagram which showed the structure of the obstacle detection apparatus schematically. It is a schematic diagram which shows the process which divides | segments the input image image | photographed with the back camera into several area | regions. It is the conceptual diagram which showed the pre-process which divides | segments a some reference image into several area | region, respectively. It is a conceptual diagram which shows the technique of the obstacle determination using the Mahalanobis distance calculated for every division area. It is a conceptual diagram illustrating an analysis example using a Mahalanobis space. It is a schematic diagram which shows the process which divides | segments another input image image | photographed with the back camera into several area | regions. It is a conceptual diagram which shows the technique of the obstacle determination using the Mahalanobis distance calculated for every division area about the input image in which an obstacle exists. It is the conceptual diagram which illustrated and showed the example of analysis using Mahalanobis space when an obstruction exists. It is a schematic diagram which shows the division area of the input image at the time of image | photographing a gravel road, and its internal analysis value. It is the schematic which illustrated the method of setting a road surface type from the input image whose road surface type is unknown. It is a schematic diagram which shows the division area of the input image at the time of imaging | photography the gravel road with an obstruction on a road surface, and its internal analysis value. It is a flowchart which shows the example of a procedure of an obstacle detection process.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic diagram showing an embodiment when an obstacle detection device is mounted on a vehicle 10. Here, the vehicle 10 is an ordinary automobile (passenger car) as an example, but an obstacle detection device may be mounted on a truck or bus. Hereinafter, the configuration of the obstacle detection device mounted on the vehicle 10 will be described.

〔camera〕
The obstacle detection device includes a rear camera 12 mounted on the rear portion of the vehicle 10. The rear camera 12 has an image sensor (CCD, CMOS, etc.), and can photograph the rear of the vehicle 10 over a wide range using, for example, a wide-angle lens. The target range photographed by the rear camera 12 includes not only the road surface R behind the currently parked vehicle 10, but also the space on the road, the distant view of the rear, and possibly the sky. it can. However, how much the shooting range (angle of view) of the rear camera 12 is set is arbitrary, and at least the road surface R up to several meters ahead and the space above it may be included in the shooting range. Further, in the vehicle width direction (horizontal direction), it is assumed that an imaging range that can cover at least the entire width of the vehicle 10 is secured.

〔Controller unit〕
The obstacle detection apparatus includes a control unit 14 together with the rear camera 12 described above. The control unit 14 has, for example, hardware resources as a microcomputer, and an in-vehicle electronic control unit (ECU) can be used as the control unit 14 as it is in this embodiment. The control unit 14 processes an image (image converted into digital data) captured by the rear camera 12 as an input image. As is well known, various in-vehicle computer hardware such as a processor, a memory device, and a peripheral IC are built in an in-vehicle ECU, so that these resources can be suitably used as the control unit 14 of the obstacle detection apparatus. it can.

〔display〕
When the obstacle detection device is applied to the vehicle 10 as in this embodiment, the vehicle-mounted liquid crystal display 16 can be used as a part of the obstacle detection device. The liquid crystal display 16 displays an input image taken by the rear camera 12 as well as various service functions (for example, a car navigation function, an AV management function, and an air conditioning management function) that are incorporated in advance in an in-vehicle ECU. Etc.) is displayed.

[Shift position detector]
The obstacle detection device can also utilize, for example, the shift position sensor 18 as a part of its configuration. The shift position sensor 18 shifts at that time according to the operation state of the shift selector lever 20 by the driver (“P”, “R”, “N”, “D”, “2”, “L”, etc.). Is output (shift position signal). In the present embodiment, detection of a stationary obstacle located behind the vehicle 10 is mainly assumed. For this reason, when it is determined based on the shift position signal that the shift position is operated to “R (reverse)” by the driver, the control unit 14 activates the imaging function and the obstacle detection function by the rear camera 12. be able to. The shift position sensor 18 can utilize the electrical components of the vehicle 10 as they are.

[Obstacle Alarm]
Further, the obstacle detection device can utilize, for example, the vehicle-mounted speaker 22 as a part of its configuration. In the present embodiment, when an obstacle located behind the vehicle 10 is detected by the obstacle detection device, an alarm sound can be output from the speaker 22 installed at the rear of the vehicle interior. Note that the electrical components of the vehicle 10 can also be used for the speaker 22 as they are.

  FIG. 2 is a functional block diagram schematically showing the configuration of the obstacle detection apparatus. In particular, FIG. 2 shows the internal configuration of the control unit 14 more specifically.

  The control unit 14 has an image input unit 24 and an input unit 26. For example, functions of peripheral ICs (for example, an input driver, an image memory, etc.) mounted on the ECU can be assigned to the input units 24 and 26. Among these, the image input unit 24 converts an image signal (for example, a serial signal) transmitted from the rear camera 12 into input image data (for example, 8-bit RGB color), and expands and stores the data in an image memory. The input unit 26 receives a shift position signal from the shift position sensor 18 and stores it as shift position information.

  The control unit 14 also has a processing unit 30. For example, a function of a processor (for example, a CPU) mounted on the ECU can be assigned to the processing unit 30. The processing unit 30 refers to the input image data stored in the image input unit 24 and executes arithmetic processing necessary for detecting an obstacle. Further, the processing unit 30 refers to the shift position information stored in the input unit 26 and recognizes the operation state (current shift position) of the shift selector lever 20 by the driver.

  The control unit 14 further has a storage unit 32. For example, a function of a memory device such as an RWM (RAM) or a ROM mounted in the ECU can be assigned to the storage unit 32. In the storage unit 32, for example, various types of data used for arithmetic processing by the processing unit 30 are stored in the storage area, and the processing unit 30 can access the storage area and read out necessary data. Various data used for the arithmetic processing will be further described later.

  The control unit 14 has an LCD controller 34. The LCD controller 34 manages a display operation by the liquid crystal display 16 based on a display signal from the processing unit 30, for example, and has a VRAM (frame buffer) corresponding to the display area of the liquid crystal display 16. Various images (for example, images taken by the rear camera 12) are displayed on the liquid crystal display 16 based on the display control signal output from the LCD controller 34. In the present embodiment, an electrical component mounted on the ECU can be used as it is for the LCD controller 34.

  In addition, the control unit 14 has an acoustic output unit 36. For example, the sound output unit 36 outputs a drive signal to the speaker 22 based on the sound output signal from the processing unit 30. Based on this drive signal, for example, the above-mentioned alarm sound (alarm sound) is emitted from the speaker 22, or sound or sound effect for car navigation, sound or music at the time of AV reproduction, etc. are emitted.

[Obstacle detection method]
Next, an obstacle detection method by the obstacle detection device will be described. FIG. 3 is a schematic diagram illustrating processing until an image captured by the rear camera 12 is used as an input image and is divided into a plurality of regions.

[Input image]
3A: For example, when the driver of the vehicle 10 operates the shift selector lever 20 to “R (reverse)”, the control unit 14 activates the photographing function of the rear camera 12, so that the rear camera 12 The rear of the vehicle 10 is photographed. The image photographed at this time includes the road surface R in the direction in which the vehicle 10 is going to travel backward (back), and the surrounding environment. In any case, the control unit 14 processes an image captured by the rear camera 12 as an input image.

  Here, as the road surface R, for example, a paved road surface in a parking lot is assumed. For this reason, most of the input image shows the asphalt surface as a dark (charcoal gray) image. In addition, a concrete block for a road shoulder, for example, appears as a whitish image on the upper right in the input image.

[Division area]
3B: The processing unit 30 of the control unit 14 performs a process of dividing the input image into a plurality of regions (for example, 16). The range to be divided at this time is set to a portion where the road surface R up to several meters behind the vehicle 10 is shown in the input image. For this reason, in FIG. 3B, the range divided in the image processing is indicated by a trapezoidal frame F, and 16 divided areas are partitioned inside. In this example, the frame F is set in a trapezoidal shape according to the range corresponding to the course when the vehicle 10 moves backward in the input image, based on the positional relationship between the rear camera 12 and the road surface R and the angle of view. However, the frame F may have a rectangular shape along the outline of the entire input image, for example.

[Pre-processing]
On the other hand, the obstacle detection device performs preprocessing based on a plurality of reference images prepared separately from the input image. Specifically, (1) capturing a plurality of reference images, (2) dividing each reference image, and (3) extracting a reference factor. Hereinafter, the pre-processing will be specifically described.

(1) Capturing a plurality of reference images Although not shown in the figure, for example, the reference image may include, for example, the same type of pavement (a road surface on which many asphalt surfaces are exposed) in a plurality of locations and in various environments (brightness and weather). , Under different time zones, etc.) and in the absence of obstacles can be used. Here, as an example, m images (for example, 10 images) captured by the rear camera 12 are taken into the control unit 14 as reference images S1 to Sm. Note that the following preprocessing including capturing of the reference image may be performed using a computer device different from the control unit 14.

(2) Division of each reference image FIG. 4 is a conceptual diagram showing pre-processing for dividing each of m reference images S1 to Sm into n regions within the frame F. In this pre-processing, each captured reference image Si (i = 1 to m) is further divided into n (16 in this example) regions Si1 to Sin. For example, the order of the divided areas proceeds in the row direction with the upper left corner in the frame F as a start point, and is folded at the right end position of each row and proceeds to the lower stage. Therefore, for example, the first reference image S1 has a divided area S11 in the upper left corner in the frame F, and a divided area S116 in the lower right corner. In other words, an arbitrary reference image Si has an arbitrary divided region Sij in the frame F, and an mth reference image Sm has a divided region Smn at the lower right corner in the frame F. Become.

  Each divided region Sij includes T pixels (for example, about 100 on average). In the present embodiment, since the divided areas Si1 to Sin are partitioned in accordance with the trapezoidal frame F, the number of pixels in the divided area increases from the upper stage to the lower stage. For example, the frame as described above It is also possible to set the number of pixels of all the divided regions Si1 to Si16 in common by making F a rectangular shape.

(3) Extraction of reference factors For each reference image Si, a plurality of types of reference factors xij and yij (in the case of two types) are extracted by calculation from T pixels included in each divided region Sij. One of the reference factors xij is a factor indicating the average luminance of the pixels in the divided region Sij, for example, and the other reference factor yij is a factor indicating the average saturation of the pixels in the divided region Sij, for example. The number of sets of a plurality of types of reference factors xij and yij extracted here is m × n for all m reference images (160 sets in this example in total).

[Average luminance: xij]
First, the average luminance xij of the pixels in the divided region Sij can be calculated from the following equation (1), for example.
xij = Σ (0.299R + 0.587G + 0.114B) / T (1)
In the above formula (1),
R: R component value of each pixel G: G component value of each pixel B: B component value of each pixel T: Total number of pixels in the divided area.

[Average saturation: yij]
Next, the average saturation yij of the pixels in the divided region Sij can be calculated from the following equation (2), for example.
yij = Σ√ [{(R−Y) /1.14} ² + {(B−Y) /2.03} ² ] / T (2)
In the above equation (2),
Y: Average luminance of the divided area Sij R: R component value of each pixel B: B component value of each pixel T: Total number of pixels in the divided area.

  The above calculation result is a set of m × n reference factors (x11, y11), (x12, y12),..., (Xmn, ymn), and from these reference factors, an average x_ave of all average luminances is obtained. The average saturation average y_ave, the average luminance variance x_sigma, the average saturation variance y_sigma, and the average luminance and average chroma covariance xy_sigma are respectively calculated.

[Average brightness: x_ave]
First, the average luminance average x_ave can be calculated from the following equation (3), for example.
x_ave = Σx / n (3)
In the above equation (3),
x: Average luminance xij (i = 1 to m, j = 1 to n)
m: reference image number (= 10)
n: number of divided areas (= 16)
It is.
[Average saturation average: y_ave]
The average y_ave of average saturation can be calculated from the following equation (4), for example.
y_ave = Σy / n (4)
In the above equation (4),
y: average saturation yij (i = 1 to m, j = 1 to n)
m: reference image number (= 10)
n: number of divided areas (= 16)
It is.

[Average luminance variance: x_sigma]
Next, the average luminance variance x_sigma can be calculated from the following equation (5), for example.
x_sigma = Σ (xx−ave) ² / n (5)
In the above equation (5),
x: Average luminance xij (i = 1 to m, j = 1 to n)
n: number of divided areas (= 16)
It is.

[Average chroma dispersion: y_sigma]
Also, the average saturation variance y_sigma can be calculated from the following equation (6), for example.
y_sigma = Σ (y−y_ave) ² / n (6)
In the above equation (6),
y: Average luminance yij (i = 1 to m, j = 1 to n)
n: number of divided areas (= 16)
It is.

The covariance xy_sigma of the average luminance and the average saturation can be calculated from the following equation (7), for example.
xy_sigma = Σ (xx−ave) · (y−y_ave) / n (7)

  It is assumed that all of the above calculation results x_ave, y_ave, x_sigma, y_sigma, and xy_sigma in the pre-processing are stored in advance in the storage unit 32 of the control unit 14.

[Static obstacle detection method]
Next, a method for detecting a stationary obstacle using an actual input image (for example, FIG. 3) will be described. The processing unit 30 of the control unit 14 can detect an obstacle through the following processing, for example.

[1] Capture of Input Image The processing unit 30 captures the input image captured by the rear camera 12 as described above. Here, it is assumed that an image of the paved road surface shown in FIG.

[2] Division of Input Image The processing unit 30 performs a process of dividing the input image into a plurality of areas. Here, it is assumed that a part of the input image is divided into n (= 16) areas in accordance with the frame F shown in FIG.

[3] Extraction of Input Factor Next, the processing unit 30 selects a plurality of types from T pixels (the number of pixels varies depending on the divided area aj) included in the divided area aj (j = 1 to n) of the input image. The input factors xj and yj are extracted by calculation. One of the input factors xj is a factor indicating the average luminance of the pixels in the divided region aj with respect to the input image, and the other reference factor yj is a factor indicating the average saturation of the pixels in the divided region aj, for example. . The right side of the above equation (1) can be applied to the calculation of the input factor xj, and the right side of the above equation (2) can be applied to the calculation of the input factor yj. The number of sets of plural types of input factors xj and yj extracted here is n (16 sets in this example in total) for the current input image.

[4] Calculation of Mahalanobis distance Next, the processing unit 30 calculates the Mahalanobis distance Dj for each divided region aj based on the set of the extracted input factors xj and yj. The Mahalanobis distance Dj for each divided area aj serves as a scale or index indicating how each divided area aj is positioned in the Mahalanobis space formed by a set of reference factors xij and yij stored in advance. It is. The Mahalanobis distance Dj can be calculated by, for example, the following calculation formula (8).

[5] Obstacle Determination for Each Divided Area FIG. 5 is a conceptual diagram showing an obstacle determination method using the Mahalanobis distance Dj calculated for each divided area aj.

[Internal analysis image]
FIG. 5A shows an example of an internal analysis image of the Mahalanobis distance Dj calculated for each divided region aj (a1 to a16). The length of the vertically long bar (bar figure) shown in each divided area aj corresponds to the Mahalanobis distance Dj for each divided area aj. As described above, in the obstacle determination process by the processing unit 30, for example, the Mahalanobis distance Dj of each divided region aj is virtually plotted on the internal memory space, and one input image is analyzed for each divided region aj. it can.

[Comparison with threshold]
In FIG. 5, (B): Here, the results of the Mahalanobis distance Dj calculated for each divided region aj are listed. On the vertical axis, a threshold value B used for obstacle determination is also shown. The determination of the obstacle by the processing unit 30 can be performed based on, for example, whether or not the Mahalanobis distance Dj of each divided region aj exceeds a threshold value B. In this example, since the Mahalanobis distance Dj does not exceed the threshold value B for any divided region aj, it can be determined that no obstacle has been detected (no obstacle) from the current input image. Note that the threshold value B can be set statistically based on many sample images taken in the absence of an obstacle.

[Mahalanobis spatial analysis]
FIG. 6 is a conceptual diagram illustrating an analysis example using the Mahalanobis space. As described above, by storing the average luminance and the average saturation as the reference factors xij and yij in the storage unit 32 in advance, a virtual Mahalanobis space (in this example, a two-dimensional space on the internal analysis by the processing unit 30). ) Can be formed.

  In such a Mahalanobis space, it is assumed that the Mahalanobis distance Dj calculated for each divided region aj for the current input image is represented by dots (“●” in FIG. 6). In the case of an input image (FIG. 3) taken with no obstacle in the target range, the analysis result using the Mahalanobis space shows that all the dots are spread in a standard group (reference symbol SG in FIG. 6). It becomes the state distributed in. This means that the Mahalanobis distance Dj is not significantly different from the standard group for any divided region aj. Therefore, in this case, it can be determined that no obstacle exists in any of the divided regions aj (does not appear in the image).

  As is well known, in the Mahalanobis space, each double ellipse representing the spread of a standard group (SG) corresponds to an equidistant line. Therefore, for example, the two dots aj1 and aj2 shown in FIG. 6 have a difference in length (Euclidean distance) in appearance, but these Mahalanobis distances are equal to each other.

[If there are obstacles]
FIG. 7 is a schematic diagram showing another input image captured by the rear camera 12 and processing until the input image is divided into a plurality of regions.

[Input image]
7A: Similarly, when the driver of the vehicle 10 operates the shift selector lever 20 to “R (reverse)”, the control unit 14 activates the imaging function of the rear camera 12 so that the rear camera 12 The rear of the vehicle 10 is photographed by the camera 12. At this time, when various obstacles BX and CN exist behind the vehicle 10, the input image includes the obstacles BX and CN together with the road surface R in the direction in which the vehicle 10 is going to move backward (back). .

  Here, for example, a cardboard box is assumed as the obstacle BX, and a road cone (pylon) is assumed as another obstacle CN. For this reason, in the input image, these obstacles BX and CN are reflected as bright and strong images. The obstacle may be a stationary object other than a cardboard box or a load cone, or may be a person.

[Division area]
7B: In any case, the processing unit 30 similarly performs processing for dividing the input image into a plurality of divided regions aj (j = 1 to n). In this example, it is shown that the entire contour of one obstacle CN is contained in one divided area, but usually the outline extends over a plurality of divided areas like an obstacle BX. Or part of the frame F protrudes.

  FIG. 8 is a conceptual diagram showing an obstacle determination method using the Mahalanobis distance Dj calculated for each divided region aj for an input image in which an obstacle exists.

[Internal analysis image]
In FIG. 8, (A): As described above, in the divided areas including the obstacles BX and CN (for example, divided areas a3, a4, a7, a10, etc.), the Mahalanobis distance is compared with other divided areas. It can be seen that it protrudes. In FIG. 8A, the portion where the vertically long bar protrudes from the image frame is indicated by a broken line.

[Comparison with threshold]
In FIG. 8, (B): In this case, since the Mahalanobis distances D3, D4, and D10 of the divided regions a3, a4, and a10 all exceed the threshold value B, in the internal analysis by the processing unit 30, these divided regions a3, a4, and a10 It can be determined that an obstacle is present (is reflected). If the Mahalanobis distance Dj does not exceed the threshold B in the other divided areas, but it is determined that an obstacle exists in any of the divided areas aj (that is, in the frame F), the processing unit 30 determines from the current input image. It is finally determined that an obstacle has been detected.

[Mahalanobis spatial analysis]
FIG. 9 is a conceptual diagram illustrating an analysis example using a Mahalanobis space when an obstacle is present. In the case of an input image (Fig. 7) taken in a state where there are obstacles (BX, CN, etc.) within the target range as in this case in the Mahalanobis space, the analysis result using the Mahalanobis space is actually an obstacle. The dots in the divided regions a3, a4, and a10 that contain the dot are in a state of protruding beyond the standard group (SG). This means that the Mahalanobis distances D3, D4, and D10 (respectively indicated by dotted arrows) for the divided areas a3, a4, and a10 are far from the standard group. Therefore, in this case, it can be determined that an obstacle exists in these divided areas a3, a4, and a10 (appears in the image).

  In FIG. 9, the dots in the divided area a10 are largely separated from the center in the horizontal axis (saturation) direction because the saturation of the obstacle CN is greatly influenced in the input image. For the other divided regions, the dots are distributed in the standard group (SG) spread.

  The above is the first example of the obstacle detection method that can be applied when it is assumed that the type of the road surface R is a paved road in advance, but due to the characteristics of the vehicle 10, the type of the road surface R is always one. However, the type of the road surface R photographed by the rear camera 12 may vary depending on the location where the vehicle 10 has moved. That is, the type of the road surface R is not limited to paved roads as the vehicle 10 moves, but may change, for example, gravel roads, farm roads, mountain roads, forest roads, and the like. If the type of the road surface R is different, the reference factor to be used for the obstacle detection is accordingly different.

  Therefore, in this embodiment, by storing reference factors for a plurality of road surface types in advance, the type of road surface R is set (estimated) based on the Mahalanobis distance calculated using an unknown input image each time. In addition, the following second example is adopted as a technique for detecting an obstacle. Hereinafter, the second example will be described.

[Second example of obstacle detection method]
FIG. 10 is a schematic diagram showing divided regions of the input image and the internal analysis values when the gravel road is photographed by the rear camera 12. That is, here, it is assumed that there are two types of road surface R, for example, “paved road” and “gravel road”, and among these, “gravel road” is taken as an input image as an example. .

[Division of input image]
10A: For example, when the driver operates the shift selector lever 20 to “R (reverse)” while the vehicle 10 is stopped (parked) on the gravel road, the rear camera 12 causes the rear of the vehicle 10 to move backward. Is filmed. At this time, in the input image, the gravel road surface (road surface R) is reflected as a grayish image in most of them, and gravel, stone blocks, etc. are scattered in the image. Further, for example, a block wall is shown as a whitish image on the upper right in the input image. The division of the input image using the frame F is the same as the processing so far.

[Internal analysis image]
FIG. 10B shows an example of the internal analysis image of the Mahalanobis distance Dj calculated for each divided region aj (a1 to a16). In particular, when there is no obstacle, the Mahalanobis distance Dj does not exceed the above threshold B even when the type of the road surface R corresponds to a gravel road.

  When it is assumed that there are a plurality of types of road surface R (for example, two types of “paved road” and “gravel road”) as in the second example, reference factors xkij, A set of ykij (k = 1, 2) is stored in the storage unit 32 of the control unit 14. That is, the storage unit 32 stores a set of reference factors x1ij and y1ij for a paved road, and also stores a set of reference factors x2ij and y2ij for a gravel road.

  As described above, since the reference factor is stored for each of the plurality of road surface types, the obstacle detection apparatus performs (1) capturing a plurality of reference images for each of the plurality of road surface types in the pre-processing, and (2). Each reference image is divided and (3) a reference factor is extracted.

  Then, the processing unit 30 calculates the Mahalanobis distance Dkj for each divided area akj for each road surface type for the current input image, and sets the road surface type closest to the center of the group in the Mahalanobis space based on the result. Can do.

[Setting example of road surface type]
FIG. 11 is a schematic diagram illustrating a method of setting a road surface type from an input image whose road surface R type is unknown.

  When the reference factors xkij and ykij are stored in the storage unit 32 for each of the plurality of road surface types as described above, the standard group (SG1, SG2) for each road surface type is included in the Mahalanobis space formed based on them. ) Spread. In this example, only the standard group (SG1) for paved roads and the standard group (SG2) for gravel roads are shown, but reference factors xkij and ykiji are stored in advance for three or more road surface types. If so, the number of standard groups corresponding to the number of road surface types (three or more) will spread in the Mahalanobis space.

[Mahalanobis distance for each road type]
In any case, when an input image whose type of road surface R is unknown is captured, the processing unit 30 first extracts the input factors xj and yj for each of the divided areas aj, and then based on these input factors xj and yj. The Mahalanobis distance Dkj is calculated for each road surface type. The Mahalanobis distance Dkj for each road surface type is calculated by substituting various variables such as average (x_ave, y_ave), variance (x_sigma, y_sigma), and covariance (xy_sigma) for each road surface type in the above equation (8). be able to.

  Therefore, for example, for a certain divided region aj in the input image, a plurality of Mahalanobis distances Dkj (k = 1, 2) are calculated for each road surface type. When this is plotted as a dot on the Mahalanobis space in FIG. 11, the Mahalanobis distance Dkj of a certain divided area (dot akj in FIG. 11) is large from the spread of the standard group (SG1) on the paved road. It can be seen that it exists in the spread of the standard group (SG2) for gravel roads. This means that this divided area (dot akj) has a shorter Mahalanobis distance for gravel roads than paved roads. Accordingly, the processing unit 30 determines that the divided region (dot akj) belongs to the standard group (SG2) for the gravel road, and “gravel road” is used as the type of the road surface R shown in the current input image. Set the road type.

  When the road surface type is set through the above processing, the processing unit 30 can determine an obstacle using the internal analysis image as before.

[If there are obstacles]
FIG. 12 is a schematic diagram showing a divided region of the input image and its internal analysis value when a gravel road with an obstacle on the road surface R is photographed by the rear camera 12. In the processing so far, it is assumed that the processing unit 30 sets the type of the road surface R to “gravel road” based on the current input image.

[Division of input image]
In FIG. 12, (A): When the obstacle BX is present behind the vehicle 10, the input image includes the obstacle BX together with the road surface R in the direction in which the vehicle 10 is going to move backward (back). For this reason, the obstacle BX is reflected in the input image as a bright and strong image. Note that the division of the input image using the frame F is performed in the same manner as the processing so far.

[Internal analysis image]
In FIG. 12, (B): The processing unit 30 reads the reference factors xij and yij for the set “gravel road” from the storage unit 32 based on the input factors xj and yj extracted from the input image, and calculates the Mahalanobis distance Dj. To do. As a result, in the divided areas a1, a2, a5, and a6 including the obstacle BX, the Mahalanobis distances D1, D2, D5, and D6 are in a state of protruding as compared with the other divided areas. When the Mahalanobis distances D1, D2, D5, and D6 exceed the threshold value B, the processing unit 30 determines that an obstacle exists (is reflected) in the divided areas a1, a2, a5, and D6. it can. Thereby, the processing unit 30 can finally determine that an obstacle has been detected from the current input image.

  As is clear from the above second example, in the present embodiment, even if the type of the road surface R is unknown, the reference factors xij and yij are stored in advance for each of a plurality of road surface types by preprocessing. First, the road surface type is set based on the input image, and then the presence or absence of an obstacle can be accurately determined based on the Mahalanobis distance calculated for each divided region.

  In the second example, “paved road” and “gravel road” are mainly assumed as road surface types, but in this embodiment, the road surface type is set from the input image after assuming more road surface types, and the final The presence or absence of an obstacle can be determined. Such an obstacle detection method can be generalized by, for example, the following processing procedure.

[Obstacle detection processing]
FIG. 13 is a flowchart illustrating a procedure example of obstacle detection processing corresponding to a large number of road surface types. For example, when the main switch of the vehicle 10 is turned on, the processing unit 30 of the control unit 14 executes this obstacle detection process periodically (for example, as an interrupt process). Hereinafter, the content of the obstacle detection process will be described along with a procedure example.

  Step S10: As the process is executed, the processing unit 30 first checks whether or not the shift position has been switched to “R (reverse)”. This confirmation can be performed based on the signal from the shift position sensor 18 described above. In particular, if the shift position is not operated to “R (reverse)” (No), the processing unit 30 returns from the obstacle detection process to the previous process (for example, the main program).

  On the other hand, when it is confirmed that the shift position is operated to “R (reverse)” (Yes), the processing unit 30 proceeds to the next step S12. The shift position can be switched from “P (parking)” → “R (reverse)” or “N (neutral)” → “R (reverse)”. Normally, the vehicle speed is 0 km / h. In addition, since the brake pedal needs to be depressed (when the shift lock is released), when the condition of step S10 is satisfied (Yes), it is considered that the vehicle 10 is currently stopped (parked).

  Step S12: Next, the processing unit 30 activates the rear camera 12. As a result, the imaging function of the rear camera 12 becomes active, and imaging of the rear (target range) of the vehicle 10 is performed.

  Step S14: When the rear camera 12 is activated, the processing unit 30 captures the current input image. The contents of the process for capturing the input image are as described together with the example of the input image shown in FIG.

(Image division processing)
Step S16: Next, the processing unit 30 executes image division processing. In this process, the processing unit 30 divides the input image into a plurality of areas. A specific division processing technique is as described together with an example of a divided area in accordance with the frame F shown in FIG.

[Extraction processing by area]
Step S18: After returning from the image segmentation process, the processing unit 30 next executes an area-specific extraction process. In this process, the processing unit 30 extracts the input factors xj and yj by calculation for each of a plurality of divided areas. Again, the right side of the above equation (1) can be applied to the calculation of the input factor xj, and the right side of the above equation (2) can be applied to the calculation of the input factor yj.

[Road surface type setting process]
Step S20: When returning from the area-specific calculation process, the processing unit 30 executes a road surface type setting process. In this process, the processing unit 30 calculates the Mahalanobis distance for each divided area based on the reference factors xij and yij for each road surface type stored in the storage unit 32, and sets the road surface type with the shortest distance among them. To do. The method of setting any one type from a plurality of road surface types is as described with the example shown in FIG. 11, for example. However, here, a road surface type setting method will be described by taking pre-processing suitable for generalization as an example.

(1) Importing multiple reference images for each road type Although not shown in the figure, in addition to the above-mentioned “paved road” and “gravel road”, “agricultural road”, “mountain road”, “forest road”, etc. A plurality of reference images obtained by capturing R types (for example, 10 types) of road surfaces in a plurality of places and in various environments and without any obstacles are captured. It is assumed that mk sheets (k = 1 to R) are prepared for each road surface type, for example, 10 sheets for “paved road” and 20 sheets for “gravel road”.

(2) Division of each reference image As a pre-process, each of the captured reference images Ski (i = 1 to m) for each road surface type is further divided into n (for example, 16) regions Ski1 to Skin. Note that the division method is as already described with reference to the example of FIG. Each divided region Skij includes T pixels (for example, about 100 on average).

(3) Extraction of reference factor for each road surface type For each reference image Ski for each road surface type, a plurality of types of reference factors xkij, ykij (in the case of two types) are calculated from T pixels included in each divided area Skij. Extract by One of the reference factors xkij is a factor indicating the average luminance of the pixels in the divided region Skij, for example, and the other reference factor ykij is a factor indicating the average saturation of the pixels in the divided region Skij, for example. The number of sets of a plurality of types of reference factors xkij and yikj extracted here is mxn for m reference images for each road surface type (for example, 160 sets of “paved road” and 320 sets of “gravel road”). Set). The average luminance xkij and the average saturation ykij can be calculated using the above equations (1) and (2), respectively.

  The above calculation results for each road surface type are set to m × n reference factor sets (xk11, yk11), (xk12, yk12),..., (Xkmn, ykmn) (k = 1 to R), and these Based on the reference factors, the average xk_ave of all average luminance, the average yk_ave of average saturation, the variance of average luminance xk_sigma, the variance of average saturation yk_sigma, and the covariance xyk_sigma of average luminance and average saturation are calculated for each road type. . These calculations can be performed using the above equations (3) to (7), respectively.

  The calculation results xk_ave, yk_ave, xk_sigma, yk_sigma, and xyk_sigma in the preprocessing are assumed to be stored in advance in the storage unit 32 of the control unit 14 for each road surface type.

(4) Calculation of Mahalanobis distance Based on the input factors xj and yij calculated for the current input image, the processing unit 30 calculates the Mahalanobis distance Dkj (k = 1 to R) for each divided region aj. For example, the following general formula (9) is used as the calculation formula, similar to the above formula (8).

  The processing unit 30 repeats the above calculation of the Mahalanobis distance Dkj k times (k = 1 to R) that is the number of road surface types, and calculates the Mahalanobis distance Dkj (j = 1 to n) for each road surface type.

(5) Road surface type setting (estimation)
Then, the processing unit 30 sets the road surface type k having the smallest value among the calculation results Dkj ² (k = 1 to R) as the type of the road surface shown in the current input image. Thereby, as a result of specifying one road surface type for the current input image, the Mahalanobis distance Dj for each divided region aj is determined again.

  Step S22: Thereby, the processing unit 30 compares the Mahalanobis distance Dj for each divided region aj with the threshold value B to determine whether there is an obstacle for each divided region aj (whether the obstacle is reflected). Can be determined. As a result, when it is determined that no obstacle exists (No), the processing unit 30 returns from the obstacle detection process to the previous process. On the other hand, when it determines with there being an obstruction (Yes), the process part 30 performs following step S24.

[Alarm output processing]
Step S24: The processing unit 30 executes an alarm output process here. In this process, the processing unit 30 issues an alarm sound output command to the sound output unit 36. In response to this, the sound output unit 36 drives the speaker 22 to actually output an alarm sound. At this time, since the warning sound is emitted from the rear position in the passenger compartment toward the driver's seat, the driver can easily recognize that an obstacle exists behind the vehicle 10 intuitively.

  At this time, the processing unit 30 may issue an alarm display output command to the LCD controller 34. Accordingly, for example, alarm information such as “attention to obstacles” is displayed on the liquid crystal display 16, thereby alerting the driver.

[Third example of obstacle detection method]
In the first and second examples of the obstacle detection method described so far, two types of factors, average luminance and average saturation, are used, but obstacle detection is performed using the following factors. May be.

  For example, as a plurality of types of factors (reference factor and input factor), R component average r_ave, G component average g_ave, B component average b_ave, RG covariance rg_sigma, GB covariance gb_sigma, BR covariance in each divided region It is assumed that six types of br_sigma are used. These factors can be calculated using, for example, the following calculation formulas (10) to (15).

r_ave = ΣR / n (10)
g_ave = ΣG / n (11)
b_ave = ΣB / n (12)
rg_sigma = Σ (R−r_ave) · (G−g_ave) / n (13)
gb_sigma = Σ (G−g_ave) · (B−b_ave) / n (14)
rb_sigma = Σ (R−r_ave) · (B−b_ave) / n (15)
In the above formulas (10) to (15),
R: R component value of each pixel G: G component value of each pixel B: B component value of each pixel.

  When the above factors are used, the Mahalanobis distance calculation formula is in accordance with the number of dimensions (in this case, 6 dimensions). Here, the above six types are given as examples of factors, but more types of factors may be used.

  As described above, according to the obstacle detection device of the present embodiment, it is reliably determined whether there is an obstacle behind the vehicle 10 based on the still image captured by the single rear camera 12. Can do. In addition, since the obstacle is determined based on a plurality of types of input factors, it is possible to increase the variation of objects that can be recognized as an obstacle.

  In this embodiment, the reference factor is extracted from the reference image obtained by photographing the road surface under various conditions in the pre-processing, and then the obstacle is based on the Mahalanobis distance (distance in the Mahalanobis space) for the current input image. Judging the presence or absence of. Therefore, the determination result is less sensitive to changes in road surface conditions (changes in brightness such as sunshine and lighting), and accordingly, erroneous determination can be prevented and obstacles can be detected with higher accuracy.

  The present invention is not limited to the embodiments described above, and can be implemented with various modifications. In one embodiment, obstacles are detected using the rear camera 12 with the rear of the vehicle 10 as a target range. However, cameras are installed in front and sides (peripheries) of the vehicle 10, respectively. An obstacle may be detected based on the input image.

  As for the road surface condition, a reference image may be prepared by finely dividing conditions such as daytime and nighttime, or when the weather is fine, cloudy, and rainy. In this case, for example, using an auxiliary device such as a brightness sensor or a road surface sensor, the road surface state (weather, time zone, dry road surface, wet road surface, etc.) of the input image is set in advance, and then a reference image is set. Obstacles can be detected.

  The obstacle detection device can also be applied to a moving body other than the vehicle 10 (for example, a railway vehicle, a towed object, a wheelchair, a cart, etc.).

  Furthermore, the target range for determining the presence or absence of an obstacle is not limited to the road surface (road surface), but may be terrain, a playground, a riverbed, a beach, a lawn, a weedy, and the like. Also in these cases, it is possible to detect an obstacle using an input image by preparing a plurality of reference images for a presumed target range and storing a reference factor for each type.

DESCRIPTION OF SYMBOLS 10 Vehicle 12 Rear camera 14 Control unit 16 Liquid crystal display 18 Shift position sensor 20 Shift selector lever 22 Speaker 30 Processing part 32 Storage part

Claims (3)

A camera for photographing a predetermined target range;
Control means for determining the presence or absence of an obstacle within the target range using an input image obtained by photographing with the camera;
The control means includes
Extracting a plurality of types of input factors for each of a plurality of divided regions obtained by dividing the input image,
A plurality of types of reference factors obtained by extracting a plurality of reference images prepared separately from the input image for each divided region obtained by dividing each of the plurality of reference images into a plurality of regions are stored in advance, and formed by these plurality of types of reference factors. A distance for each divided region of the input image in the reference space to be calculated based on the plurality of types of input factors,
An obstacle detection device that determines the presence or absence of an obstacle within the target range for each divided region of the input image based on the calculated distance. The obstacle detection device according to claim 1,
The control means includes
A Mahalanobis space formed by the plurality of types of reference factors is used as the reference space, and a Mahalanobis distance for each divided region of the input image in the Mahalanobis space is calculated as a distance in the reference space.
An obstacle detection apparatus that determines the presence or absence of an obstacle within the target range for each divided region of the input image based on the Mahalanobis distance. In the obstacle detection device according to claim 1 or 2,
The control means includes
For each of a plurality of road surface types included in the target range, a plurality of reference images prepared separately from the input image are divided into a plurality of regions, and a plurality of types obtained by extracting for each of the divided regions. A reference factor is stored in advance for each of the plurality of road surface types, and a distance for each divided region of the input image in the reference space is calculated based on the plurality of types of input factors for each of the plurality of road surface types. Set the road surface type included in the target range based on the distance
An obstacle detection device that determines the presence or absence of an obstacle in the target range for each of the divided areas based on the distance calculated for the set road surface type.