JP2014179052A - Image processor, image processing method, image processing program, image processing system and mobile device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor for detecting a travel area imaged in an image with high accuracy.SOLUTION: The image processor has frequency calculation means for calculating a frequency of a parallax of a small area every horizontal position of an image about a parallax calculated by parallax calculation means, inter-boundary small area calculation means for calculating an inter-boundary small area being a prescribed small area located between two boundary areas detected by boundary area detection means, and a travel area detection means for detecting a travel area from the inter-boundary small area toward a small area of a horizontal end part in the image on the basis of the frequency.

Description

  The present invention relates to an image processing device, an image processing method, an image processing program, an image processing system, and a moving device.

  2. Description of the Related Art Conventionally, development of a technique for detecting a three-dimensional object that obstructs movement in movement of a vehicle, a ship, an aircraft, an industrial robot, or the like (hereinafter referred to as a vehicle) has been advanced. In particular, a technique for recognizing a road on which a vehicle or the like travels and determining whether a three-dimensional object exists on the road is known.

  Patent Document 1 describes a technology for recognizing a road shape by separating only white lines on an actual road using three-dimensional position information. The technique described in Patent Document 1 includes two-dimensional information representing an image and distance data to an imaged object by processing an image captured by a stereo image processing apparatus having two CCD cameras. Get 3D information. In the space represented by the three-dimensional information, the white line is on the road plane, while the three-dimensional object such as the preceding vehicle is at a position higher than the road plane. In this way, the stereo image processing apparatus distinguishes whether it is a road or a three-dimensional object according to the height from the white line based on the acquired three-dimensional information.

  Further, in a captured image that is generally captured by an imaging device mounted on a vehicle or the like, buildings, guardrails, walls, and the like (hereinafter referred to as buildings) are captured on the left and right of the captured image. In the technique described in Patent Literature 2, a traveling region is detected by detecting a portion corresponding to a wall such as a building based on parallax calculated from a plurality of captured images. Here, the traveling area refers to an area between buildings and the like that are imaged on the left and right.

However, when the road surface is curved to the right with respect to the traveling direction, as shown in FIG. 24, it is assumed that the travel region in which the vehicle or the like can travel is on the right as the distance from the imaging device increases.
Even if it is assumed that the traveling area is on the right, if an attempt is made to detect the traveling area from all the areas in the image, for example, due to noise or the like, A traveling region may be detected near the upper left corner. Similarly, when the road surface is curved to the left with respect to the traveling direction, it is assumed that the traveling region is on the left side as the distance from the imaging device increases. Nevertheless, if a white line is to be detected in all the areas in the image, the running area may be detected in the area near the upper right end in the image.
As described above, when it is determined whether or not all areas in the image are travel areas, an erroneous determination occurs, and there is a problem that the travel area captured in the image cannot be detected with high accuracy. ing.

  In order to solve the above-described problem, in the present invention, an image processing apparatus that processes a plurality of images in which a plurality of boundaries are captured, a boundary region detection unit that detects a boundary region captured in the image, A parallax calculation unit that calculates parallaxes of the small areas constituting the plurality of images, and a parallax frequency of the small area is calculated for each position in the horizontal direction of the image with respect to the parallaxes calculated by the parallax calculation unit. A frequency calculation means, a boundary small area calculation means for calculating a small area between boundaries that is a predetermined small area between two boundary areas detected by the boundary area detection means, and And a travel region detecting unit configured to detect a travel region based on the frequency toward a small region at a horizontal end in the image.

According to the present invention, for the parallax calculated by the parallax calculating means, the frequency calculating means for calculating the frequency of the parallax of the small area for each position in the horizontal direction of the image, and the two boundary areas detected by the boundary area detecting means A small inter-boundary region calculating means for calculating a small inter-boundary region that is a predetermined small region between the boundary and the frequency from the small inter-boundary region toward the small region at the end in the horizontal direction in the image. And the travel region detecting means for detecting the travel region based on the travel region detection means, the travel region in the image can be determined with high accuracy.

1 is a schematic diagram illustrating an example of a vehicle or the like on which an image processing system 1 is mounted. 2 is a hardware configuration diagram of the image processing system 1. FIG. 1 is a functional configuration diagram of an image processing system 1. FIG. It is an example of the captured image which the imaging part imaged. It is a figure for _{demonstrating} parallax _dxy . FIG. 4 is an enlarged view of a part A of a reference image Ia, a figure showing luminance for each small area of part A, and a figure showing a luminance difference for each small area of part A. Is a diagram showing a correspondence relationship between the parallax d _e in the y-coordinate and its position representative of the position of the white line edge. It is a figure used in order to calculate the frequency C from the reference | standard image Ia. It is a frequency distribution diagram showing the distribution of the frequency C in the reference image Ia. 3 is a flowchart showing an outline of processing of the image processing system 1. FIG. 6 is a flowchart showing processing of a signal conversion unit 52. FIG. 10 is a flowchart showing a first process of an edge candidate detection unit 55. FIG. 12 is a flowchart for explaining an outline of second processing of the edge candidate detection unit 55. FIG. 12 is a flowchart showing details of second processing of the edge candidate detection unit 55. FIG. It is the figure which expanded a part A of the reference | standard image Ia. It is a flowchart showing the process of the curve direction determination part 58. FIG. FIG. 11 is a flowchart showing processing of a straight intersection detection unit 581. FIG. 10 is a flowchart showing processing of a vanishing point detection unit 582. FIG. 10 is a flowchart showing processing of white line parallax determination unit 56. FIG. 10 is a flowchart showing processing of a travel region detection unit 57. It is a flowchart showing the process of the frequency determination part 572. 4 is another example of a functional configuration diagram of the image processing system 1. FIG. FIG. 6 is a diagram showing a luminance difference for each small region of a part A of a reference image Ia and a diagram showing a parallax for each small region of part A. It is an example of the captured image which the imaging part imaged.

  An embodiment of the present invention will be described with reference to FIGS.

[Hardware Configuration of Image Processing System 1]
First, the hardware configuration of the image processing system 1 in the embodiment of the present invention will be described. FIG. 1 is a conceptual diagram showing an automobile equipped with an image processing system 1 according to the present embodiment. FIG. 1A is an overview of a side surface of an automobile that is an example of a vehicle or the like in which an image processing system 1 is provided. FIG. 1 (2) is a schematic view of the front of the automobile shown in FIG. 1 (1). FIG. 2 is a hardware configuration diagram of the image processing system 1.

  As shown in FIGS. 1 (1) and (2), the image processing system 1 includes an imaging device 10a and an imaging device 10b, and the imaging device 10a and the imaging device 10b capture a scene in front of the traveling direction of the automobile. It is installed as follows. As illustrated in FIG. 2, the image processing system 1 includes an imaging device 10 a, an imaging device 10 b, a signal conversion device 20 a, a signal conversion device 20 b, and an image processing device 30. The imaging device 10a captures a front scene and generates an analog signal representing a captured image, and includes an imaging lens 11a, a diaphragm 12a, and an image sensor 13a.

  The imaging lens 11a is an optical element for forming an image of the subject S by refracting light passing through the imaging lens 11a. The diaphragm 12a adjusts the amount of light input to the image sensor 13a described later by blocking part of the light that has passed through the imaging lens 11a. The image sensor 13a is a semiconductor element that converts light input from the imaging lens 11a and the aperture 12a into an electrical analog image signal, and is realized by a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor). The

  The imaging device 10b has the same configuration as the imaging device 10a. Therefore, the description about the imaging device 10b is omitted. The imaging lens 11a and the imaging lens 11b are installed so that the imaging surface related to the imaging lens 11a and the imaging surface related to the imaging lens 11b are in the same plane.

  The signal conversion device 20a converts an analog signal representing a captured image into captured image data in a digital format, such as a CDS (Correlated Double Sampling) 21a, an AGC (Auto Gain Control) 22a, an ADC (Analog Digital Converter) 23a, And a frame memory 24a.

  The CDS 21a removes noise from the analog image signal converted by the image sensor 13a by correlated double sampling. The AGC 22a performs gain control for controlling the intensity of the analog image signal from which noise has been removed by the CDS 21a. The ADC 23a converts the analog image signal whose gain is controlled by the AGC 22a into captured image data in a digital format. The frame memory 24a stores the captured image data converted by the ADC 23a.

  The signal conversion device 20b acquires captured image data from an analog image signal converted by the imaging device 10b, and includes a CDS 21b, an AGC 22b, an ADC 23b, and a frame memory 24b. The CDS 21b, AGC 22b, ADC 23b, and frame memory 24b have the same configuration as the CDS 21a, AGC 22a, ADC 23a, and frame memory 24a, respectively, and thus description thereof is omitted.

  The image processing device 30 is for processing the captured image data converted by the signal conversion device 20a and the signal conversion device 20b. The image processing device 30 is an FPGA (Field Programmable Gate Array) 31, a CPU (Central Processing Unit) 32, and a ROM ( It has a Read Only Memory (33) and a RAM (Random Access Memory) 34. The FPGA 31 performs a process of calculating the parallax d in the captured image represented by the captured image data. The CPU 32 controls each function of the image processing apparatus 30. The ROM 33 stores an image processing program that the CPU 32 executes to control each function of the image processing apparatus 30. The RAM 34 is used as a work area for the CPU 32.

  The image processing program may be recorded in a computer-readable recording medium and distributed as a file in an installable or executable format.

[Functional Configuration of Image Processing System 1]
Next, a functional configuration of the image processing system 1 of the present embodiment will be described. FIG. 3 is a functional configuration diagram of the image processing system 1. The image processing system 1 includes an imaging unit 51, a signal conversion unit 52, a parallax calculation unit 53, a distance calculation unit 54, an edge candidate detection unit 55, a white line parallax determination unit 56, a travel region detection unit 57, and a curve direction determination unit 58. doing.

  FIG. 4 is an example of a captured image captured by the imaging unit 51. The imaging unit 51 generates a scene in front of the traveling direction of the automobile as shown in FIG. 4A as a captured image, and is realized by the imaging device 10a and the imaging device 10b. The signal conversion unit 52 converts an analog signal representing a captured image generated by the imaging unit 51 into digital captured image data, and is realized by the signal conversion device 20a and the signal conversion device 20b.

  The parallax calculation unit 53, the distance calculation unit 54, the edge candidate detection unit 55, the white line parallax determination unit 56, the travel area detection unit 57, and the curve direction determination unit 58 are realized by the CPU 32 according to a program stored in the ROM 33.

In the description of the present embodiment, coordinates as shown in FIG. 4B are used to represent the position in the captured image of the subject S captured in the captured image. One unit of coordinates shown in FIG. 4 may be composed of one pixel, or may be composed of two or more adjacent pixel groups. A portion applied in black indicated by coordinates (x, y) in FIG. 4B is referred to as one small region P _xy composed of one or two or more adjacent pixel groups. In the following description of the configuration and operation of each functional unit, each functional unit performs processing on each small region P _xy . In the captured image, the y coordinate shown in FIG. 4B is information representing the position in the image in the direction perpendicular to the road surface.

<< Parallax calculation unit >>
The parallax calculation unit 53 calculates the parallax d _{xy of the} subject S captured in each small area P _xy of the captured image. Here, the parallax d _xy will be described with reference to FIG. FIG. 5 is a diagram for explaining the parallax d _xy . In the following description, the captured images captured by the imaging device 10a and the imaging device 10b shown in FIG. 5 are referred to as a reference image Ia and a comparative image Ib, respectively. Further, the position of each small region P _{xy in} the captured image is represented by coordinates (x, y) (x = 1 to 1280, y = 1 to 960) as shown in FIG. In the reference image Ia, the x-axis direction shown in FIG. 4B is referred to as the horizontal direction of the image, and the y-axis direction is referred to as the vertical direction of the image. In the reference image Ia, the direction in which the x coordinate is increased is referred to as the right direction, and the direction in which the x coordinate is decreased is referred to as the left direction. In the reference image Ia, the direction in which the y coordinate is increased is referred to as an upward direction, and the direction in which the y coordinate is decreased is referred to as a downward direction.

In FIG. 5, when the distance from the imaging center of the imaging lens 11a at the imaging position Sa of the subject S is Δa and the distance from the imaging center of the imaging lens 11b at the imaging position Sb of the subject S is Δb, The parallax d _xy is d _xy = Δa + Δb. That is, the parallax d _xy is the coordinates (x, y) of the position where the subject S is imaged as shown in FIG. 5 in the reference image Ia and the coordinates (x, y) of the position where the subject S is imaged in the comparison image Ib. x ′ difference (x′−x) in x ′, y).

The parallax calculation unit 53 calculates the parallax d _xy for each coordinate (x, y) of the small area P _xy included in the predetermined small area group in the reference image Ia. Information that associates the coordinates (x, y) and the parallax d _xy is referred to as parallax image data. The predetermined small area group is a set of small areas P _xy other than the small area P _xy related to the subject S captured in both the reference image Ia and the comparative image Ib. For example, among the subjects S captured near the left end of the reference image Ia, there are subjects S that are not captured in the comparative image Ib because they exceed the imaging range of the comparative image Ib. For such a small region P _xy related to the subject S, the coordinate x ′ cannot be recognized, and the parallax d _xy cannot be calculated. Such a small area P _xy is not included in a predetermined small area group.

<< Distance calculation section >>
The distance calculation unit 54 calculates the distance Z using the parallax d _xy calculated by the parallax calculation unit 53. The distance Z is a distance from the surface parallel to the imaging surfaces of the imaging lens 11a and the imaging lens 11b to the subject S including the center of the imaging lens 11a and the center of the imaging lens 11b. Specifically, as shown in FIG. 5, the focal length f of the imaging lens 11a and the imaging lens 11b, the baseline length B that is the length between the center of the imaging lens 11a and the center of the imaging lens 11b, and the parallax d , The distance calculation unit 54 calculates the distance Z = (B × f) / d. The baseline length B and the focal length f may be 5 to 30 (cm) and 4 to 15 (mm), respectively, as an example.

Here, the relationship between the distance Z and the parallax d will be further described. From the calculation formula of Z = (B × f) / d, the larger the parallax d, the smaller the distance Z, and the smaller the parallax d, the larger the distance Z. For example, if the parallax d _kl at the coordinates (k, l) in FIG. 4 (2) is larger than the parallax d _{jl at} the coordinates (j, l), the distance Z _{kl at} the coordinates (k, l) is The value is smaller than the distance Z _{jl at the} coordinates (j, l).

That is, it is determined that the subject at the coordinates (k, l) (the car in FIG. 4 (2)) is closer to the subject at the coordinates (j, l) (white line in FIG. 4 (2)). . In this manner, the parallax d _xy in each small region P _xy, by comparing the parallax d _w in the small area in the image white line is captured (. As white regions P _w), each small area P _xy It is possible to determine whether or not represents a road surface.

<< Edge candidate detection unit >>
Edge candidate detection unit 55, the white line of the edge on the road surface being imaged on the reference image Ia (hereinafter referred white line edge) small regions P _xy candidate representing a (hereinafter, referred to as the white line edge candidate region P _e) detecting the . Here, the white line is a part of an image representing a white line drawn on a general road surface. That is, the white line in the present embodiment is not a one-dimensional line but a band-like portion having a width in the x-axis direction.

  The function of the edge candidate detection unit 55 will be described with reference to FIG. FIG. 6 is an enlarged view of a part of the reference image Ia. FIG. 6A is an enlarged view of a portion A which is a part of FIG. 4A, and a portion represented in white represents a part of a white line in the reference image Ia and is represented by a diagonal line. The portion to be represented represents a part of the road surface other than the white line in the reference image Ia.

FIG. 6B is a diagram illustrating the luminance L of the pixels constituting each small area P _xy of the captured image illustrated in FIG. As shown in FIG. 6B, the luminance of the pixels in the small area P _xy that forms part of the white line is higher than the luminance of the pixels in the small area P _xy that forms part of the road surface other than the white line. . FIG. 6 (3) is a diagram showing the luminance difference ΔL of the pixels constituting each small region P _xy shown in FIG. 6 (2). In the present embodiment, the description will be made assuming that the luminance L takes a value from 0 to 255.

The edge candidate detection unit 55 determines a difference in luminance L between two small regions P _xy adjacent to each other in the x-axis direction for each small region P _xy constituting the reference image Ia as shown in FIG. ΔL) is calculated. Specifically, the edge candidate detection unit 55 _converts the luminance L (x + 1, y) of the small area P _{xy at} the coordinates (x + 1, y) from the luminance L (x, y) of the small area P _{xy at} the coordinates (x, y). The luminance difference ΔL (x, y) = L (x, y) −L (x + 1, y) obtained by subtracting y) is calculated.

In this way, the calculated luminance difference [Delta] L (x, y) small is shown in FIG. 6 (3), represented by P _e in the small region P _e representing the edge portion of the white line _(Fig. 6 (3) The luminance difference ΔL (x, y) in the region is larger than that in other portions.

The edge candidate detection unit 55 determines whether or not the calculated luminance difference ΔL (x, y) is a predetermined threshold, for example, 120 or more. Then, the edge candidate detection unit 55 when the luminance difference [Delta] L (x, y) is determined as 120 or more, detects the small region _{P xy} at the coordinates (x, y) as a white line edge candidate area _{P e.}

In the case where small region P _xy is composed of two or more pixels, the edge candidate detection unit 55, if the small area P _xy luminance L and the average value of the luminance L of the pixel group making up the small region P _xy Good. Further, the edge candidate detection unit 55 calculates a luminance difference ΔL with respect to an adjacent pixel for each pixel, and uses the small region P _xy including the pixel determined to have a luminance difference ΔL of 120 or more as the white line edge candidate region _Pe. It may be detected.

In addition, the edge candidate detection unit 55 in the small region P _xy group having a constant y coordinate of the reference image Ia has a small left end from the central small region P _xy (the small region located at x = 640 in FIG. 4 (2)). The above calculation and determination are sequentially performed toward the region P _xy . Further, the edge candidate detection unit 55 ends the calculation and determination processing for the small region P _xy group related to the y coordinate when the left white line edge candidate region P _el is detected. Therefore, in the diagram shown in FIG. 6 (3), the luminance difference ΔL (x, y) for the small region P _xy located on the left side of the small region P _{xy having} the luminance difference ΔL (x, y) of 120 or more is Not calculated.

Similarly, the edge candidate detection unit 55 has a small right end from the central small region P _xy (the small region located at x = 640 in FIG. 4 (2)) in the small region P _xy group having a constant y coordinate of the reference image Ia. The above calculation and determination are sequentially performed toward the region P _xy . Further, the edge candidate detection unit 55 detects the right white line edge candidate region _Per .

<< White Line Parallax Determination Unit >>
White disparity determination unit 56 based on the white line edge candidate area P _e which is determined by the edge candidate detection unit 55, determines a white line disparity d _w is the disparity of the white line area P _w. As shown in FIG. 3, the white line parallax determination unit 56 includes a regression line calculation unit 561 and a determination unit 562.

<Regression line calculation unit>
Regression line calculation section 561 calculates a regression line of y-coordinate and the parallax d _e of the white line edge candidate region P _e detected by the edge candidate detection unit 55. Here, the function of the regression line calculation unit 561 will be described with reference to FIG. Figure 7 is a diagram showing a correspondence relationship between the parallax d _e in the y-coordinate and its position representative of the position of the white line edge candidate region P _e. Y-coordinate on the vertical axis in FIG. 7, the parallax d _e is shown on the horizontal axis, and the y-coordinate of the white line edge candidate area P _e group, corresponding disparity d _e is represented by dots. For these points, the regression line calculation unit 561 calculates a regression line Reg-Line by Hough transformation, which is a known regression line calculation method.

<Determining part>
Determination unit 562 determines the disparity _{d e} in the regression line Reg-Line as white disparity _{d w} is the disparity in the white line area _{P w.}

<< Running area detection unit >>
The travel area detection unit 57 detects the travel area from the small area P _xy that forms the reference image Ia. Buildings in the reference image Ia, guard rail, a wall or the like (hereinafter, referred to as a building, etc.) When is a small region being imaged and object region P _b, the left side of the object region P _bl and right object region P _br the running region Is a small region group between the two. The travel region detection unit 57 includes a frequency calculation unit 571, a frequency determination unit 572, and a detection unit 573.

<Frequency calculator>
The frequency calculation unit 571 calculates the frequency C of the parallax d _xy calculated by the parallax calculation unit 53. The degree C, the disparity d _xy among the small regions P _xy represented in the reference image Ia by the coordinate x is the number of small regions P _xy which is a predetermined value.

Here, the function in which the frequency calculation unit 571 calculates the frequency C will be described using a specific example with reference to FIGS. 8 and 9. FIG. 8 is a diagram used for calculating the frequency C from the reference image Ia, and FIG. 9 is a frequency distribution diagram showing the distribution of the frequency C in the reference image Ia. FIG. 8A shows coordinates representing the position of each small region P _{xy with} respect to the reference image Ia shown in FIG. In this specific example, the small area P _xy is set larger than the example shown in FIG. 4B for easy explanation, but the size of the small area P _xy may be set as appropriate.

FIG. 8B is a diagram showing the parallax d _xy for a part of the small area P _xy of the reference image Ia shown in FIG. For example, in the small area P _xy of the coordinates (3, 1) to the coordinates (3, 4) shown in FIG. 8 (1), the subject S is the road surface and has the same height. The distance Z from the imaging devices 10a and 10b increases as the y coordinate increases for the small region P _xy of the coordinates (3, 1) to the coordinates (3, 4). Therefore, it is shown that the parallax d _xy is smaller than the equation of Z = (B × f) / d described above with the base line length B and the focal length f being constant. In addition, since the subject S is a building wall in the small area P _xy of the coordinates (3, 5) to the coordinates (3, 10), the distance Z from the imaging devices 10a and 10b is the same even if the y coordinate increases. Therefore, it is indicated that the parallax d _xy is almost the same. Further, regarding the coordinates (3, 11) and the coordinates (3, 12), since the subject S is empty, the distance Z is approximated to infinity, so that the parallax d _xy is almost zero. FIG. 9 is a diagram showing the frequency C of the parallax d _{xy in} the small region P _xy at the coordinate x with the coordinate x shown in FIG. 8 (2) on the horizontal axis and the parallax d _xy on the vertical axis.

When the frequency calculation unit 571 calculates the frequency C for the reference image Ia illustrated in FIG. 8A, the parallax d _xy is included in the small area P _xy where x = 3 as illustrated in FIG. 8B. There are six that are 100 (see W1 in FIG. 8 (2)). Therefore, the frequency calculation unit 571 represents the frequency C “6” corresponding to (x, d _xy ) = (3, 100) on the frequency distribution diagram shown in FIG. Similarly, since there is one each of parallax d _xy of 120, 115, 110, and 105 among the small regions P _xy with x = 3, the frequency calculation unit 571 has (x, d _xy ) = (3,120), (3,115), (3,110), and (3,105), the frequency C “1” is represented.

Similarly, as shown in FIG. 8 (2), among the small regions P _xy with x = 4, there are five that have parallax d _xy of 95 (see W2 in FIG. 8 (2)). The frequency calculation unit 571 calculates the frequency C as “5”. Therefore, the frequency calculation unit 571 represents the frequency C “5” corresponding to (x, d _xy ) = (4, 95) on the frequency distribution diagram shown in FIG. 9. Further, as shown in FIG. 8 (2), since there are five small areas P _xy with x = 15 and the parallax d _xy is 80 (see W3 in FIG. 8 (2)), the frequency calculation is performed. The unit 571 calculates the frequency C as “5”. Therefore, the frequency calculation unit 571 represents the frequency C “5” corresponding to (x, d _xy ) = (15, 80) on the frequency distribution diagram. In this way, the frequency calculation unit 571 calculates the frequency C for the x coordinate of the small area P _xy included in the predetermined small area group.

<Frequency determination unit>
The frequency determination unit 572 determines whether the frequency C calculated by the frequency calculation unit 571 is greater than or equal to a predetermined threshold Cp. Specifically, the frequency determining unit 572 displays the left white line edge candidate region P _el detected by the edge candidate detection unit 55 x coordinate on the horizontal axis, the frequency distribution diagram showing the parallax d _xy on the vertical axis. Then, the left white line edge candidate region displayed below the predetermined parallax d _xy is linearly approximated to calculate an approximate straight line (see line L3 in FIG. 9). Further, the left white line edge candidate region displayed below the predetermined parallax d _xy (predetermined parallax d _xy = 100 in the present embodiment) is approximated by a curve to calculate an approximate curve (see line L4 in FIG. 9). .

This function will be described with reference to the examples shown in FIGS. For example, the small area P _xy represented by the coordinates (3, 2) is detected by the edge candidate detecting unit 55 as the left white line edge candidate area P _el . As shown in FIG. 8 (2), since the parallax d _xy = 115 of the small region P _xy represented by the coordinates (3,2), the frequency determination unit 572 has coordinates x = 3 and d _{xy of the} frequency distribution diagram. = 115 is displayed indicating that it is the left white line edge candidate area _Pel (see (v) of FIG. 9).

Similarly, the small area P _xy represented by the coordinates (4, 3) is detected by the edge candidate detection unit 55 as the left white line edge candidate area _Pel . As shown in FIG. 8B, since the parallax d _xy of the small region P _xy represented by the coordinates (4, 3) is 110, the frequency determination unit 572 has coordinates x = 4 and d _{xy of the} frequency distribution diagram. = 110 is displayed indicating that it is the left white line edge candidate region _Pel (see the mark (vi) in FIG. 9).

  Similarly, all the left white line edge candidate areas are displayed in the frequency distribution diagram, and the approximate straight line is calculated as line L3 shown in FIG.

Similarly, the frequency determination unit 572 displays the right white line edge candidate region _Per detected by the edge candidate detection unit 55 on a frequency distribution diagram in which the horizontal axis represents the x coordinate and the vertical axis represents the parallax d _xy . Then, the left white line edge candidate area displayed below the predetermined parallax d _xy is linearly approximated to calculate an approximate straight line R3. Further, the approximated line lineR4 is calculated by approximating the left white line edge candidate area displayed below the predetermined parallax d _xy with a curve.

Further, the frequency determination unit 572 calculates a straight line (hereinafter referred to as a bisector lineh3) that approximates a small area that is in the middle of the small areas corresponding to the same y coordinate in the approximate straight line lineL3 and the approximate straight line lineR3. Further, the frequency determination unit 572 calculates a straight line (hereinafter referred to as a bisector lineh4) that approximates a small area that is in the middle of the small area corresponding to the same y coordinate in the approximate straight line L4 and the approximate straight line R4. Further, a mathematical expression (x = f (d _xy )) representing the line lineh including the bisector lineh3 and the bisector lineh4 is calculated.

From the coordinates (f (d), d) on the frequency distribution map corresponding to the line segment lineh, the coordinates (1, d _xy ) and the coordinates (x _max , d _xy ) of the end of the frequency distribution chart are sequentially displayed. It is determined whether the frequency C corresponding to x, d _xy ) is equal to or greater than a predetermined threshold Cp. Here, x _max is the maximum value of the x coordinate in the reference image Ia and the frequency distribution diagram, and in the specific example shown in FIG. 9, x _max = 16.

When it is determined by the frequency determination unit 572 that C ≧ Cp, the frequency determination unit 572 indicates that the small area P _xy on the reference image Ia corresponding to the coordinates (x, d _xy ) in the frequency distribution diagram represents a building or the like It is said. When the frequency determination unit 572 determines that C ≧ Cp is not satisfied, the frequency determination unit 572 does not represent a building or the like on the reference image Ia corresponding to the coordinates (x, d) in the frequency distribution diagram.

<Detector>
Based on the determination by the frequency determination unit 572, the detection unit 573 detects a travel region from among the small regions P _xy captured in the reference image Ia. As described above, the frequency C of the small area P _xy where the subject S is a building or the like is larger than the frequency C of the small area P _xy where the subject S is a road surface. That is, when the frequency C is equal to or greater than the predetermined threshold Cp, the small region P _xy related to the frequency C is a building or the like, and thus the small region P _xy is detected as the object region P _b in the reference image Ia. Detector 573 detects the object area P _bl and right object region P _br left imaged in the reference image Ia, detects the small region P _xy which lies between these as the travel area.

<< Curve direction determination unit >>
The curve direction determination unit 58 determines whether the road surface imaged in the reference image Ia is curved in the right direction or curved in the left direction. The curve direction determination unit 58 includes a straight intersection detection unit 581, a vanishing point detection unit 582, and a determination unit 583.

In the reference image Ia in which a curved road surface is generally imaged, as shown in FIG. 4 (2), the white line in the front portion as viewed from the imaging device 10, that is, the predetermined y ≦ y _S portion is a straight line. (Hereinafter referred to as a straight line portion). Further, in such standard image Ia, the back portion Te imaging apparatus 10 viewed from, ie white line portion of a given y> y _S is curvilinear (hereinafter, referred to as curved portion).

<Linear intersection detection unit>
Straight intersection point detection unit 581 detects the y ≦ _y left white line edge candidate region _{P el} of _S, the intersection _{P i} of the respective approximate lines of the right white line edge candidate area _{P er.} Specifically, straight intersection point detection unit 581 linearly approximates the left white line edge candidate area _{P el} expressed in terms of FIG. 4 (2), and calculates an approximate straight line LineL1. Similarly, the linear intersection point detection unit 581 is a linear approximation the right white line edge candidate area _{P er} represented in terms of FIG. 4 (2), and calculates an approximate straight line LineR1. Further, the linear intersection detection unit 581 calculates coordinates (x _i , y _i ) representing a small area that is an intersection of the approximate straight line L1 and the approximate straight line R1 and detects it as a straight intersection P _i .

<Vanishing point detector>
Vanishing point detecting unit 582 represents the approximate curve lineL2 according to the left of the white line edge candidate region _{P el} predetermined y> _{y S,} the intersection become small area between the approximate curve lineR2 according to the right of the white line edge candidate region _{P er} The coordinates (x _v , y _v ) are calculated, and the intersection is detected as the vanishing point P _v . The vanishing point detection unit 582 is an example of a vanishing point detection unit.

<Determining unit>
The determination unit 583 determines the direction in which the road surface is curved based on the vanishing point P _v detected by the vanishing point detection unit 582 and the intersection P _i detected by the linear intersection detection unit 581. Specifically, the determination unit 583 compares whether the x-coordinate _{x v} of the vanishing point _{P v} is the x-coordinate _{x i} is smaller than the intersection _{P i,} is identical to the greater or _{x i.}

If x _v > x _i by this comparison, the curve direction determination unit 58 determines that the road surface is curved to the right. If x _v <x _i , the curve direction determination unit 58 determines that the road surface is curved to the left. Further, when it is determined that x v ₌ x _i, the curve direction determination unit 58 determines that the road surface is not curved.

The white line is an example of a boundary, the white line region _Pw is an example of a boundary region, and the edge candidate detection unit 55 is an example of a boundary region detection unit. The parallax calculation unit 53 is an example of a parallax calculation unit. The travel area detection unit 57 is an example of a travel area detection unit. The frequency calculation unit 571 is an example of a frequency calculation unit. The frequency determination unit 572 is an example of a frequency determination unit.

[Operation of Image Processing System 1]
Subsequently, the operation of the image processing system 1 according to the present embodiment will be described with reference to FIGS. 10 to 21. FIG. 10 is a flowchart showing an outline of processing of the image processing system 1. FIG. 11 is a flowchart showing the processing of the signal conversion unit 52. FIG. 12 is a flowchart showing the first process of the edge candidate detection unit 55. FIG. 13 is a flowchart for explaining the outline of the second process of the edge candidate detection unit 55. FIG. 14 is a flowchart showing details of the second processing of the edge candidate detection unit 55. FIG. 15 is an enlarged view of a part A of the reference image Ia. FIG. 16 is a flowchart showing the processing of the curve direction determination unit 58. FIG. 17 is a flowchart showing the processing of the straight intersection detection unit 581. FIG. 18 is a flowchart showing the processing of the vanishing point detection unit 582. FIG. 19 is a flowchart showing the processing of the white line parallax determination unit 56. FIG. 20 is a flowchart showing the processing of the travel area detection unit 57. FIG. 21 is a flowchart showing the processing of the frequency determination unit 572.

<< Captured Image Generation Processing >>
As shown in FIG. 10, first, the imaging unit 51 captures a scene in front of the vehicle in the traveling direction and generates captured images at predetermined time intervals (step S1). Specifically, the amount of light is adjusted by refracting the light passing through the imaging lens 11a that realizes the imaging unit 51 and blocking a part of the light that the diaphragm 12a has passed through the imaging lens 11a. The light input from the imaging lens 11a is imaged, and the image sensor 13a converts the imaged image into an electrical analog image signal. The imaging lens 11b, the diaphragm 12b, and the image sensor 13b operate in the same manner as the imaging lens 11a, the diaphragm 12a, and the image sensor 13a, respectively.

<< Process for signal conversion >>
Subsequently, the signal converter 52 converts the analog image signal into digital captured image data (step S2). Specifically, as shown in FIG. 11, the CDS 21a first performs correlated double sampling on the analog image signal to remove noise (step S21). Then, the AGC 22a performs gain control for controlling the intensity of the analog image signal from which noise has been removed (step S22). When the analog image signal is gain-controlled by the AGC 22a, the ADC 23a converts the analog image signal into digital captured image data (step S23). When the ADC 23a converts an analog image signal into captured image data, the frame memory 24a stores the captured image data. The CDS 21b, AGC 22b, and ADC 23b operate in the same manner as the CDS 21a, AGC 22a, and ADC 23a, respectively.

<< Process for calculating parallax >>
Returning to FIG. 10, next, the parallax calculation unit 53 calculates the parallax d _xy for the small area P _xy included in the predetermined small area group in the reference image Ia represented by the captured image data (step S3).

<< Process for detecting edge candidates >>
Next, as shown in FIG. 10, the edge candidate detection unit 55 detects a white line edge candidate region P _e of the reference image Ia (Step S4). In the reference image Ia, the edge candidate detection unit 55 determines the luminance difference ΔL (x, y) = L (x, y) −L (with respect to each pixel in the small region P _xy adjacent in the x-axis direction for each small region P _xy. x + 1, y) is calculated. [Delta] L (x, y) is equal to or greater than a predetermined threshold, the edge candidate detection unit 55 detects the small region P _xy as the edge candidate region P _e.

In reference image Ia already commonly road as described is imaged, front portion as viewed from the imaging device as shown in FIG. 4 (2), i.e., the white line of the following moiety predetermined y coordinate y _s is A straight line is formed (hereinafter, this portion is referred to as a straight portion). Further, in such standard image Ia, the rear portion as viewed from the imaging device, namely a white line of a given y-coordinate y _s larger portion is curvilinear (hereinafter, this portion of the curved portion). Note that y _s is a value determined from the results of experiments, and when traveling on a curved road, it is preferable that the white line is processed as a straight line in a small region where the y coordinate is y _s or less. Value. Y _s is it is preferable that a value near the middle of the y-axis direction of the reference image Ia as an example.

Will be described in <first detection processing of the edge candidate region>, edge candidate detection unit 55 detects a white line edge candidate region P _e of the linear portion of the white line. For the white line curve portion, a white line edge candidate region is detected by <second candidate edge region detection process> described later for the purpose of improving accuracy.

In the description of these processes, the position of each small region P _xy in the reference image Ia is coordinates (x, y) (x = 1 to 1280, y = 1 to 960) as shown in FIG. It is assumed that

<First detection process of edge candidate area>
In the first detection process, as shown in FIG. 12, first, the edge candidate detection unit 55 adds 1 to y = 0 (step S41), y = 1 (step S42), x = 641 (step S43). ) Minus the luminance difference ΔL (640,1) = L (640,1) − for the luminance L (640,1) of the pixel in the small area P _xy at the coordinate according to x = 640 (step S44). L (641, 1) is calculated (step S45). Then, the edge candidate detection unit 55 determines whether or not the luminance difference ΔL (640, 1) calculated in step S45 is 120 or more (step S46). If it is determined in step S46 that ΔL (640, 1) is not 120 or more, the x coordinate related to the luminance difference is subtracted by 1 (step S44). The luminance difference ΔL (639) related to the small area P _xy in the coordinates. , 1) = L (639,1) −L (640,1), the edge candidate detection unit 55 calculates (step S45). Then, the edge candidate detection unit 55 determines whether or not the luminance difference ΔL (639, 1) is 120 or more (step S46).

Furthermore, when it is determined in step S46 that ΔL (639,1) is not 120 or more, the luminance of the small region P _{xy at} the coordinate is obtained by subtracting 1 from the x coordinate related to ΔL (639,1) (step S44). The edge candidate detection unit 55 calculates the difference ΔL (638,1) = L (638,1) −L (639,1) (step S45). Then, the edge candidate detection unit 55 determines whether or not the luminance difference ΔL (638, 1) is 120 or more (step S46). In this way, the calculation (step S45) and determination (step S46) of the luminance difference ΔL related to the small area P _xy at the coordinate obtained by subtracting the x coordinate by 1 are repeated, and it is determined that the luminance difference ΔL is 120 or more. When, for detecting a small area _{P xy} according to the luminance difference ΔL as white line edge candidate area _{P e} (step S47).

Further, if the white line edge candidate region P _e is detected, it determines the edge candidate detection unit 55 whether or not y ≧ y _s for the y-coordinate (step S48).

If the y coordinate of the white line edge candidate region _{P e} is determined not y ≧ _{y s} in step S418, 1 is added to y coordinate (step S42), obtained by subtracting 1 from x = 641 (step S43) (Step S44) The luminance difference ΔL (640,2) = L (640,2) −L (641,2) related to the coordinates (640,2) is calculated (Step S45). Then, the processing from step S42 to step S48 is repeated as described above. Repeat these processing, and ends the process of detecting the y ≧ y _s a is determined as When the white line edge candidate region P _e of the process.

In this manner, by calculating the luminance difference ΔL sequentially for each small region P _xy reference image Ia, it can be calculated white line edge candidate region P _e for each y-coordinate.

As described in the description of the processing in steps S43 to S46, the small area P _xy (coordinate (640, y)) in the central portion of the reference image Ia to the outer small area P _xy (coordinate (1, y)). The edge candidate detection unit 55 calculates and determines the luminance difference ΔL in order toward (), and ends this process when the white line edge candidate region _Pe is detected. That is, these processes for the small areas P _xy of x-coordinate is smaller than the x coordinate of the white line edge candidate region P _e is not performed. For this reason, it is possible to reduce the load on the CPU 32 that implements the edge candidate detection unit 55.

  When the processing up to step S48 is completed, a luminance difference ΔL (x, y) is calculated for each coordinate (x, y) as shown in FIG.

Incidentally, the processing described with reference to FIG. 12 aims to detect the white line edge candidate region P _e being imaged to a small area on the left from the center in the reference image Ia (coordinate x = 1-640) is there. Therefore, where the detected white-line edge candidate region P _e is the left white line edge candidate area P _el representing the left white line edge than the center. Further, a white line is drawn not only on the left side but also on the right side on a general road surface on which the vehicle travels. For this reason, the edge candidate detection unit 55 performs the same processing on the small region (coordinates x = 641 to 1280) on the right side of the center in the reference image Ia. Then, the detected white line edge candidate region P _e is the right white line edge candidate region P _er.

<Second Detection Process of Edge Candidate Area (Overview)>
The second detection process of the edge candidate area will be described with reference to FIGS. FIG. 13 is a flowchart for explaining the outline of the second process of the edge candidate detection unit 55. FIG. 14 is a flowchart showing details of the second processing of the edge candidate detection unit 55. FIG. 15 is an enlarged view of a part A of the reference image Ia.

Here, before describing the details of the second detection process, it will be described that the result of determination by the curve direction determination unit 58 is used in the second detection process. As already described, the imaging unit 51 generates captured images at predetermined time intervals (step S1 in FIG. 10). Since the vehicle or the like on which the image processing system 1 is mounted at the predetermined time moves by a predetermined distance in the traveling direction, the sequentially captured images are different from each other. As shown in FIG. 13, for the captured image generated by the imaging unit for the first time (step S <b> 1-1), the edge candidate detection unit 55 performs white lines as described in <First detection process of edge candidate region> above. detecting an edge candidate region P _e of the edge candidate region P _e and the curved portion of the white line of the linear portion of the (step S4-1). Then, was detected on the basis of these edge candidate region P _e, the curve direction determining unit 58 determines the direction in which the road surface is curved (step S6-1). Details of the process performed by the curve direction determination unit 58 in step S6-1 will be described later.

Imaging unit has generated the second time for (step S1-2) captured image, detects the edge candidate region P _e of the linear portion of the white line as described in the <first detection processing of the edge candidate region> ( Step S4-2). Then, based on the curve direction determined by the curve direction determination unit 58 for the first time, the edge candidate region of the curve portion of the white line is detected as described later in <Second candidate edge region detection processing (details)>. (Step S5-2). Subsequently, using each of these edge candidate regions _Pe , the curve direction determination unit 58 determines the direction in which the road surface is curved (step S6-2).

Similarly, for the captured image generated by the imaging unit at the nth time (n ≧ 2) (step S1-n), the edge of the straight line portion of the white line as described in the above <first detection process of edge candidate region> Candidate area _Pe is detected (step S4-n). Then, based on the direction determined by the curve direction determination unit 58 in the (n−1) th time, as described later in <Second detection process of edge candidate area (details)>, the edge candidate area of the curve portion of the white line _Pe is detected (step S5-n). Then, using each of these edge candidate regions _Pe , the curve direction determination unit 58 determines the direction in which the road surface is curved (step S6-n).

<Second Detection Process of Edge Candidate Area (Details)>
Next, details of the second detection process performed by the edge candidate detection unit 55 in step S5 will be described with reference to FIGS. Here, first, the detection process when it is determined that the road surface of the captured image captured before a predetermined time before the captured image related to the second detection process is curved to the right will be described. To do. Above, the white line edge candidate region _{P e} at y = _{y s} in the first detection processing has already been detected, in the description here to the coordinates _{(x s, y} s) _(FIG. 15 (i) Small area indicated by

First, the edge candidate detection unit 55 adds y to y = y _s (step S51), y = y _s +1 (step S52), and x = x _s −1 (step S53) adds 1 to x = x _s. The luminance difference ΔL (x _s , y _s +1) = L () for the luminance L (x _s , y _s +1) of the pixel in the small region P _xy (see FIG. 15 (ii)) at the coordinates related to (Step S54) x _{s, y s +1)} calculates the _{-L (x s + 1, y} s +1) ( step S55). Then, the edge candidate detection unit 55 determines whether or not the luminance difference ΔL (x _s , y _s +1) calculated in step S55 is 120 or more (step S56). In step _{S56 ΔL (x s, y s} +1) is the case where it is determined not to be 120 or more, obtained by adding 1 to the x-coordinate of the luminance difference (step S54) small regions _{P xy} at the coordinates (Fig. 15 luminance difference ΔL according to see _{(iii)) (x s +} 1, y s +1) = L (x s + 1, y s +1) -L (x s + 2, y s +1) edge candidate detection unit 55 is calculated (Step S55). Then, the edge candidate detection unit 55 determines whether or not the luminance difference ΔL (x _s1 +1, y _s +1) is 120 or more (step S56).

Further, [Delta] L in step _{S56 (x s + 1, y} s +1) is the case where it is determined not to be 120 or more, obtained by adding 1 to the x-coordinate of the _{ΔL (x s + 1, y} s +1) ( step S54) The edge candidate detection unit 55 calculates the luminance difference ΔL (x _s +2, y _s +1) = L (x _s +2, y _s +1) −L (x _s +3, y _s +1) of the small region P _{xy at} the coordinates. (Step S55). Then, the edge candidate detection unit 55 determines whether or not the luminance difference ΔL (x _s +2, y _s +1) is 120 or more (step S56). In this way, the calculation (step S55) and determination (step S56) of the luminance difference ΔL related to the small region P _xy at the coordinate obtained by adding the x coordinate by 1 are repeated, and the luminance difference ΔL (x _s + n, y _s +1) is repeated. ) is if it is determined that 120 or more, to detect a small region _{P xy} according to the luminance difference ΔL as white line edge candidate area _{P e} (step S57).

When the luminance difference ΔL of the white line edge candidate region P _e is determined to be 120 or more in step S57, y _v edge or not is whether or candidate detection unit 55 is a y coordinate of the y-coordinate is the vanishing point P _v Is determined (step S58).

If the y coordinate of the white line edge candidate region _{P e} is determined not to be _{y v} or more in step S58, the 1 is added to y coordinate (step _S52), (a _{= x s1) x = x s} + n coordinates _{(x s1,} y s +2) small regions _{P xy} in (FIG. 15 (iv)) luminance difference ΔL according to _{see) (x s1, y s +2} ) = L (x s1, y s +2) - L (x _s1 +1, y _s +2) is calculated (step S55). Then, the processing from step S52 to step S58 is repeated in the same manner as described above. Repeat these processes, y coordinates terminates the process for detecting a white line edge candidate region P _e if it is determined that y _v or more.

If the road surface being imaged is curved to the right in the reference image Ia generated before a predetermined time, x-coordinate of the white line edge candidate region P _e in the captured image increases with increasing y coordinate Therefore, as described above, the luminance difference ΔL is calculated and determined while increasing the x coordinate as the y coordinate increases. That is, whether already or brightness difference ΔL to detect the white line edge candidate region P _e for the upper right small area P _xy than the small region P _xy which white line edge candidate region P _e is detected is the predetermined value or more Processing to determine is performed. Such by performing the process in the order it is possible to reduce the erroneous detection of the white line edge candidate region P _e, to improve the accuracy of the process for determining the road surface.

Similarly, when it is determined that the road surface imaged in the reference image Ia generated a predetermined time before is curved to the left, the small coordinate at the coordinate obtained by subtracting the x coordinate by 1 in step S54. The calculation (step S55) and determination (step S56) of the luminance difference ΔL related to the region P _xy are repeated, and if the luminance difference ΔL is determined to be 120 or more, the small region P _xy related to the luminance difference ΔL is determined as a white line edge candidate. detecting a region _{P e} (step S57).

When the road surface is detected white line edge candidate region P _e while decreasing the x-coordinate with the increase of the y-coordinate if the curved to the right, is likely to be erroneously detected that the white line edge candidate region P _e. Therefore, by the calculation and determination of the luminance difference ΔL with increasing x-coordinate with the increase of the y-coordinate, it is possible to reduce the erroneous detection of the white line edge candidate region P _e, the accuracy of a process for determining the road surface improves.

In the case where the road surface is curved to the left, since the x-coordinate of the white line edge candidate region P _e in the captured image is to be reduced with increasing y coordinate, the x coordinate with the increase of the y-coordinate, as described above The luminance difference ΔL is calculated and determined while decreasing the.

<< Process for judging curve >>
Returning to FIG. 10, the edge candidate detection unit 55 detects a white line edge candidate region P _e (step S4 and step S5), and the curve direction determining unit 58 road surface is captured in the reference image Ia is to right It is determined whether the curve is curved or curved in the left direction (step S6).

<Process for detecting intersection of approximate lines>
First, as shown in FIG. 16, the straight intersection detection unit 581 calculates an intersection P _i between the extension line of the left white line on the reference image Ia and the extension line of the right straight line (step S61).
Specifically, (see FIG. 4 (2)) by linearly approximating the left approximation line lineL1 straight intersection point detection unit 581 for the white line edge candidate region P _el group of the linear portion of the left side as shown in FIG. 17 Is calculated (step S611). Similarly, the linear intersection point detection unit 581 linearly approximates the white line edge candidate area _{P er} groups of the linear portion of the right side, and calculates an approximate straight line LineR1 (see Figure 4 (2)) (step S612). The linear intersection detection unit 581 then calculates coordinates (x _i , y _i ) representing the position of the intersection P _i between the right approximate line line L1 and the approximate line line R1 (step S613). Note that either the process of calculating the left approximate line line L1 or the process of calculating the right approximate line line R1 may be performed first.

<Disappearance detection process>
Subsequently, returning to FIG. 16, the vanishing point detection unit 582 detects the vanishing point _Pv of the left and right white lines (step S62). Specifically, calculates the approximate curve vanishing point detecting unit 582 for the white line edge candidate region _{P el} group left curved portions as shown in FIG. 18 LineL2 (see Figure 4 (2)) (Step S621) . Similarly, the vanishing point detecting unit 582 calculates an approximate curve LineR2 (see Figure 4 (2)) for white line edge candidate area _{P er} groups of the right curved portion (step S622). Further, the vanishing point detection unit 582 calculates coordinates (x _v , y _v ) representing the position of the vanishing point P _v that is the intersection of the approximate curve line L2 and the approximate curve line R2 (step S623). Note that either the process of calculating the left approximate curve lineL2 or the process of calculating the right approximate curve lineR2 may be performed first.

<Judgment process>
Then, again returning to FIG. 16, based on the vanishing point _{P v} and the intersection _{P i} of the approximate line lineR1 the approximate line LineL1, road surface judging unit 583 is captured in the reference image Ia is curved to the right in the direction Or whether it is curved in the left direction (step S63). Specifically, when the coordinates of the intersection point P _i are (x _i , y _i ), the determination unit 583 determines whether x _v that is the x coordinate representing the position of the vanishing point P _v is larger or smaller than x _i . Or x _v = x _i . If it is determined that x _v is greater than x _i as shown in FIG. 4 (2), the determination unit 583 determines that the road surface is curved to the right. Further, when the x _v is determined that x _i is smaller than, the determination unit 583 determines that the road surface is curved to the left. Further, when it is determined that x v ₌ x _i, the determination unit 583 determines that the road surface is linear.

<< Process for determining white line parallax >>
Subsequently, returning to FIG. 10, the white line parallax determining section 56 determines the disparity d _w of the white line area P _w based on the white line edge candidate area P _e detected by the edge candidate detection unit 55 in step S4 and step S5 (Step S7). Specifically, as shown in FIG. 19, first, a regression line calculation section 561 calculates a regression line of y-coordinate and the parallax d _e of the white line edge candidate region P _e (step S71). Specifically, for each y-coordinate and the parallax _{d e} of the white line edge candidate region _{P e,} white disparity determination unit 56 determines the regression line Reg-line shown in FIG. 7 by the Hough transform. Then, the white line parallax determining section 56 determines the disparity _{d e} on the regression line Reg-line at each y-coordinate and the left white line parallax _{d w} of the y-coordinate (step S72).

Can remove noise contained in the white line edge candidate area P _e by the reference image Ia is calculated in this manner regression line Reg-line, it is possible to determine the white line disparity d _w with high accuracy. Thereby, the distance Z can be calculated accurately.

<< Process to detect travel area >>
Subsequently, returning to FIG. 10, the travel region detection unit 57 detects the travel region from the small region P _xy included in the predetermined small region group of the reference image Ia (step S8).

<Process to calculate frequency>
First, the small number calculating unit 571 as shown in FIG. 20 is a small area for P _xy, the disparity d _xy among the small regions P _xy represented by coordinates x of the reference image Ia included in a predetermined small region groups The frequency C of the area P _xy is calculated (step S81). As described above, the result calculated by the frequency calculation unit 571 is shown in FIG.

<Process to determine frequency>
The frequency determination unit 572 determines whether or not the frequency C calculated by the frequency calculation unit 571 is greater than or equal to the threshold value Cp (step S82). Power C of the frequency distribution diagram power C is the subject S of the frequency distribution diagram according to the small region P _xy object S is a building or the like as described in the <number calculating section> is according to the small region P _xy which is a road surface Greater than. That is, it can be determined whether or not the small area P _xy represents a building or the like by determining whether or not the frequency C is larger than a predetermined threshold Cp.

Here, details of a process in which the frequency determination unit 572 determines whether the frequency C is greater than a predetermined threshold Cp will be described with reference to FIG. For frequency determination unit 572 is a process of determining, approximation in the frequency distribution diagram of first white line edge candidate area P _e groups of the left and right of the boundary between the small area calculating unit 574 is detected by the edge candidate detection unit 55 linearly lineL3 A line lineh (see FIG. 9) including a bisector lineh3 of the approximate line lineR3 and a bisector lineh4 of the approximate curve lineL4 and the approximate curve lineR4 is calculated (step S821). Specifically, the inter-border small region calculation unit 574 displays the x coordinate and the parallax d of each of the left white line edge candidate region _Per and the right white line edge candidate region _Per in the frequency distribution diagram. Then, the boundary between the small area calculating unit 574 approximates the left white line edge candidate area _{P el} linear lineL3 and right white line edge candidate region _{P er} approximate straight line LineR3, and bisector lineh3 thereof as shown in FIG. 9 calculate. Also, the boundary between the small area calculating unit 574 left white line edge candidate region _{P el} approximate curve lineL4 and right white line edge candidate trendline area _{P er lineR4,} and their bisector lineh4 as shown in FIG. 9 calculate. In the following description, it is assumed that a line including the bisector lineh3 and the bisector lineh4 is lineh, and lineh is expressed by the equation x = f (d) on the frequency distribution diagram. Note that the small area P _xy corresponding to lineh is an example of a predetermined small area, and the small boundary area calculation unit 574 is an example of a small boundary area calculation unit.

Then, the frequency determination unit 572 displays the frequency C of the coordinates (f (d), d) (see FIG. 9 (i)) on the line segment lineh on the frequency distribution diagram represented by the coordinates (x, d). (Step S822), it is determined whether or not C ≧ Cp (step S823). When it is determined in step S823 that C ≧ Cp, the frequency determination unit 572 represents a small area P _xy on the reference image Ia corresponding to the coordinates (x, d) in the frequency distribution diagram in which a building or the like is represented. (Step S824). When it is determined in step S823 that C ≧ Cp is not satisfied, the frequency determination unit 572 subtracts 1 from the coordinate x in the frequency distribution chart (step S825) coordinates (f (d) -1, d) (FIG. 9 (ii) It is determined whether or not C ≧ Cp with respect to the frequency C according to (see step S823).

Further, when it is determined that C ≧ Cp is not satisfied, the frequency determination unit 572 subtracts 1 from the coordinate x in the frequency distribution chart (step S825) coordinates (f (d) −2, d) (see FIG. 9 (iii)). It is determined whether or not C ≧ Cp for the frequency C related to () (step S823). In this manner, the frequency C is determined while sequentially subtracting 1 from the x coordinate (step S825) (step S823), and the coordinates (f (d) -n, d) (see FIG. 9 (iv)) are determined. If it is determined that C ≧ Cp for the frequency C, the frequency determination unit 572 indicates that a building or the like is represented in the small area P _xy corresponding to the coordinate x = f (d) −n and the parallax d. Determination is made (step S824).

  Here, a process in which the frequency determination unit 572 determines whether or not the frequency C is equal to or greater than a predetermined threshold will be described using a specific example of the frequency distribution diagram illustrated in FIG. 9. In this specific example, it is assumed that the predetermined threshold is Cp = 4. For example, the processing for the parallax d = 100 in the frequency distribution diagram is as follows. The frequency determination unit 572 calculates the x coordinate corresponding to the parallax d = 100 of the bisector lineh on the frequency distribution diagram as x = f (100) (step S822). In the specific example shown in FIG. 9, x = f (100) = 9. At this time, the frequency determination unit 572 determines whether or not C ≧ Cp for the frequency C at the coordinates (x, d) = (9, 100) (see FIG. 9I) (step S823). In this specific example, since C = 1 and Cp = 4, it is determined that C ≧ Cp is not satisfied.

Next, the frequency determination unit 572 subtracts 1 from the coordinate x (step S825). Whether or not C ≧ Cp for the frequency C at the coordinates (x, d) = (8, 100) (see FIG. 9 (ii)). Is determined (step S823). This process is repeated to determine whether C ≧ Cp for the frequency C at coordinates (x, d) = (3, 100) (see FIG. 9 (iv)) (step S823), C = 6. Since Cp = 4, it is determined that C ≧ Cp. Thereby, the frequency determination unit 572 determines that the small area P _xy corresponding to the coordinate x = 3 and the parallax d = 100 is a building or the like (step S824) and ends the process. If it is determined that the building is a building or the like, it is not determined whether or not the frequency C at the coordinates x = 1 to 2 and the parallax d = 100 is C ≧ Cp.

  Although the processing related to the parallax d = 100 has been described above, the same processing is performed for all the parallaxes d on the frequency distribution diagram. In the above description, the frequency C with respect to the coordinates (x, d) to the left of the line segment lineh is the target of processing. However, the processing in step S825 is performed by adding 1 to the coordinate x in the frequency distribution diagram. It is similarly determined whether or not the frequency C with respect to coordinates (x, d) to the right of lineh is a building or the like.

<Process to detect>
Detector 573 detects the small region P _xy which is between the object region P _br and right object region P _br left detected by the frequency determining unit 572 as the travel area.

  In this way, whether or not a building or the like is in order from the frequency C with respect to the coordinates (x, d) on the line segment lineh determined in the frequency distribution diagram from the coordinates (1, d) at the end in the x-axis direction. The process which determines is performed. If it is determined that the frequency C related to a certain coordinate (x, d) is a building or the like, the frequency C related to the coordinate (x, d) at the end portion is not determined from the coordinate (x, d). . Accordingly, for example, the portion B shown in FIG. 4A, that is, the left portion when the curve direction is right, is not determined as to whether it is a road surface. Therefore, erroneous detection due to noise or the like can be prevented, and the road surface imaged in the image can be detected with high accuracy.

[Supplement of Embodiment]
In the above embodiment, the frequency determination unit 572 sequentially sets the coordinates (x, d) from the coordinates (f (d), d) corresponding to the line segment lineh on the frequency distribution chart toward the end coordinates (1, d). ) Is determined whether or not the frequency C corresponding to () is equal to or greater than the threshold value Cp. However, instead of the line segment lineh, the coordinates (x, d) correspond to the coordinates (x, d) in order from the predetermined coordinates (x, d) between the approximate line line L3 and the approximate line line R3 toward the end coordinates (1, d). It may be determined whether or not the frequency C is greater than or equal to the threshold Cp.

Further, the above-described determination may be performed based on the vanishing point _Pv detected by the vanishing point detection unit 582. Specifically, a line segment lineh ′ connecting the small area P _xy located in the center of the small areas P _xy related to the largest d _xy coordinates in the frequency distribution diagram and the vanishing point P _v is used instead of line h. . Since the x-coordinate of the vanishing point P _v vanishing point P _v from being present at infinity by and vanishing point detecting section 582 that is d = 0 is detected, the frequency as shown in FIG. 9 The vanishing point position _Pv on the distribution chart can be represented on the frequency distribution chart, and lineh ′ can be calculated based on this.

In addition, when the curve direction is determined by the determination unit 583, the above-described determination may be performed according to the curve direction. Specifically, when the curve direction determination unit 58 determines that the road surface is curved in the right direction, the line segment lineh ″ passing through the vanishing point P _v represented on the frequency distribution map and parallel to the line L3 is obtained. And use this instead of lineh. Although a line segment parallel to lineL3 is used here, the present invention is not limited to this, and a line segment having an inclination in the curve direction can be widely used. Further, when the curve direction determination unit 58 determines that the road surface is curved in the right direction, the line segment lineh '' passing through the vanishing point _Pv represented on the frequency distribution map and parallel to the lineR3 is calculated, This is used instead of lineh.

Thus power determination unit 572, since the power C based on the vanishing point P _v to determine whether it is greater than a predetermined value Cp, there can approximate curve lineL4 the approximate curve lineR4 is not accurately calculated In addition, the travel area detection unit 57 can efficiently detect the travel area.

In the determination based on the vanishing point, the line segment calculated by the vanishing point and the small region _Pxy having the largest d _xy coordinate in the frequency distribution diagram is used, but the small region P having the largest _dxy coordinate is used. _The small area P _xy in which the d _xy coordinate in the frequency distribution diagram has a predetermined value may be used for calculating the line segment line ′. The predetermined value here is the maximum parallax d _xy determined empirically.

As represented in the parallax distribution diagram of FIG. _9, in the _{d xy} coordinate is the maximum subregion _{P xy} there may be no lineL3, LineR3. In such a case, even if the frequency determination unit 572 makes the determination already described, the load on the CPU 32 that implements the determination unit 572 increases. Therefore, in such a case, the frequency determination unit 572 can efficiently process only the small region P _{xy in} which the d _xy coordinates are less than a predetermined value. As a result, it is possible to reduce the load on the CPU 32 of the image processing apparatus.

Each in the order corresponding to the coordinates of the end portion from the coordinates of the approximate line LineL3 left white line edge candidate pixels rather than the line segment lineh _{P el} (1, d) a headed sequentially each coordinate (x, d) the coordinates (x , D), it may be determined whether or not the frequency C corresponding to or higher than the threshold Cp. It is assumed that the small area P _xy related to the coordinates (x, d) between the coordinates in the frequency distribution diagram of the left white line edge candidate area P _ele and the right white line edge candidate area P _er is not a building or the like. Therefore, a process is performed to determine whether or not only the frequency C of coordinates (x, d) not between the left white line edge candidate region _Pel and the right white line edge candidate region _{Per in} the frequency distribution diagram is equal to or greater than the threshold Cp. For example, the load on the CPU can be reduced.

FIG. 22 is another example of a functional configuration diagram of the image processing system 1. In the above embodiments, by having a white line reliability determining unit 563 as the white line parallax determining section 56 shown in FIG. 22, likelihood white line edge candidate region P _e is a small region representing the white line edge (hereinafter, white It is also possible to determine the accuracy). Here, a process in which the white line accuracy determination unit 563 determines the white line accuracy will be described with reference to FIG. Figure 23 is similar to the view shown in FIG. 6, of the white line edge candidate area P _e group detected by the edge candidate detection unit 55, the one white line edge candidate region P _e, and the white line edge candidate region P _e The three small regions P _e1 , P _e2 , and P _e3 adjacent to are denoted by reference numerals. FIG. 23 is an enlarged view of a part of the reference image Ia.

First, when the white line reliability determining unit 563 edge candidate region _{P e} shown in FIG. 23 (1) in step S4 or step S5 is detected, the white line edge candidate region _{P e} and x-axis direction shown in FIG. 23 (2) The parallax is compared for three small regions P _e1 , P _e2 , and P _e3 that are adjacent to each other outside. Specifically, the parallax _{d e} of the white line edge candidate region _{P e,} small regions _{P e1,} P _e2, the difference between the disparity of _{P e3 d e1, d e2,} d e3 each _{d e -d} e1, _d e - d _e2 , d _e −d _e3 (hereinafter referred to as parallax differences Δd ₁ , Δd ₂ , and Δd ₃ ) are calculated. Then, it is determined whether each of the parallax differences Δd ₁ , Δd ₂ , Δd ₃ is a predetermined value, for example, 10 or less. In the example shown in FIG. 23 (2), since d _e = 132, d _e1 = 130, d _e2 = 128, and d _e3 = 132, Δd ₁ = 132−130 = 2 and Δd ₂ = 132−128 =. 4, Δd ₃ = 132−132 = 0. Then, it is determined whether or not these values are not more than a predetermined value, that is, not more than 10.

Parallax difference [Delta] d _1, [Delta] d _2, if none is [Delta] d ₃ is less than a predetermined value, the white line reliability determining unit 563 determines that the white line edge candidate region _{P e} is the white line area _{P w.} Also, if the disparity difference [Delta] d _1, [Delta] d _2, one of [Delta] d ₃ is not a predetermined range, the white line reliability determining unit 563 determines that the white line edge candidate region _{P e} is not a white line area _{P w.} In the example shown in FIG. 23, the parallax difference [Delta] d ₁ as described above, [Delta] d _2, [Delta] d ₃ is the white line reliability determining unit 563 because it is either 10 or less, the white line edge candidate region _{P e} is a white line area _{P w} Is determined. Furthermore, these determination performed for all of the white line edge candidate region P _e, the regression line calculation section 561 performs a process for calculating a regression line using only the edge candidate region P _e which is determined to be the white line area P _w .

The white line on the road surface is a band-shaped portion having a width in the x-axis direction. Therefore, parallax disparity _{d e1, d e2, d e3} in white line edge candidate area _{P e} from the predetermined width (three small regions _{P xy} In the above example adjacent to the x-axis direction) in the white line edge candidate region _{P e} d _{When the} value is close to _e , it indicates that the subjects S represented by these small regions P _xy are close to each other. That is considered white line edge candidate region P _e is a high probability is representative of the white line present on the road surface. Therefore, the travel area detection unit 57 by such accuracy is used only high white line edge candidate region P _e becomes possible to perform more accurate road determined.

In the above example with three small areas adjacent, used for the determination was determined the accuracy of the white line edge candidate region P _e, the angle of view of the imaging lens 11a and an imaging lens 11b, the resolution, the type of the white line It is preferred to determine the number of subregions to be made.

In the present embodiment, in the small region P _xy group having a constant y coordinate of the reference image Ia, the edge candidate detection unit 55 sequentially changes the luminance difference ΔL from the central small region P _xy toward the leftmost small region P _xy . calculated and it is to perform the determination, it may perform the calculation and determination of the luminance difference ΔL toward the center of the small region P _xy at the left end of the small region P _xy to a predetermined number of small regions P _xy intervals.

  Further, in the present embodiment, processing for detecting a white line edge is performed, but the object to be detected here is not a white belt-shaped portion, but is used to represent a boundary between a roadway and a sidewalk or a boundary between lanes on a road surface. It may be a belt-like portion of other colors.

In the <first detection process of edge candidate area> of the present embodiment, the edge candidate detection unit 55 starts from the small area P _xy (small area located at the x coordinate 640) in the center of the reference image Ia in the x-axis direction. The luminance difference ΔL is calculated and the determination based on the luminance difference ΔL is sequentially performed toward the left end and the right end small area P _xy . However, the edge candidate detection unit 55 may perform calculation and determination sequentially from other small regions P _xy instead of the central small region P _xy . It is preferable to appropriately set so as to perform the calculation and determination so as to detect early edge candidate region P _e in accordance with the features of the reference image Ia.

  In the present embodiment, the imaging device 10, the signal conversion device 20, and the image processing device 30 are configured as one casing. However, an independent stereo camera as another housing may include the imaging device 10 and the signal conversion device 20, and the image processing device 30 may be configured as another housing.

  In the present embodiment, the image processing system 1 is mounted on a vehicle. However, the image processing system 1 may be mounted on a mobile device such as a vehicle, a ship, an aircraft, or an industrial robot in addition to the vehicle.

DESCRIPTION OF SYMBOLS 1 Image processing system 10 Imaging device 11 Imaging lens 12 Aperture 13 Image sensor 20 Signal processing part 21 CDS
22 AGC
23 ADC
24 frame memory 30 image processing device 31 DSP
32 CPU
33 ROM
34 RAM
DESCRIPTION OF SYMBOLS 50 Image processing part 51 Image pick-up part 52 Signal conversion part 53 Parallax calculation part 55 Edge candidate detection part 56 White line parallax determination part 561 Regression straight line calculation part 562 Determination part 57 Travel area detection part 571 Frequency calculation part 572 Frequency determination part 573 Detection part 58 Curve direction determination unit 581 Straight line intersection detection unit 582 Vanishing point detection unit 583 Determination unit

JP-A-5-265547 Proceedings of the fifth international Conference on 3D Digital Imaging and Modeling 1550-6185 / 05 $ 20.00 2005 IEEE

Claims (11)

An image processing apparatus that processes a plurality of images obtained by imaging a plurality of boundaries,
Boundary region detection means for detecting a boundary region captured in the image;
Parallax calculating means for calculating parallax of small areas constituting the plurality of images;
A frequency calculation means for calculating the parallax frequency of the small area for each horizontal position of the image with respect to the parallax calculated by the parallax calculation means;
An inter-boundary small area calculating means for calculating an inter-boundary small area that is a predetermined small area between two boundary areas detected by the boundary area detecting means;
The running area detecting means for detecting the running area based on the frequency from the small area between the boundaries toward the small area at the end in the horizontal direction in the image;
An image processing apparatus comprising: The image processing apparatus according to claim 1,
The image processing apparatus according to claim 1, wherein the small area between the boundaries is a midpoint between the two boundary areas having the same vertical position. The image processing apparatus according to claim 1,
The image processing apparatus according to claim 1, wherein the small area between the boundaries is the boundary area. An image processing apparatus according to any one of claims 1 to 3,
A frequency determination means for determining whether the frequency calculated by the frequency calculation means is equal to or greater than a predetermined value;
The small area determined by the frequency determination means that the frequency is greater than or equal to a predetermined value is defined as an object area,
An image processing apparatus, wherein a small area between a left object area and a right object area in the image is detected as the travel area. An image processing apparatus according to any one of claims 1 to 4,
The boundary area detecting means includes
An image processing apparatus, wherein the boundary region is detected based on a luminance of a pixel forming the image. An image processing apparatus according to any one of claims 1 to 5,
The boundary area detecting means includes
An image processing apparatus that detects a small region having a pixel having a luminance difference between adjacent pixels equal to or greater than a predetermined threshold as a candidate for a boundary region. The image processing apparatus according to any one of claims 1 to 6,
Vanishing point detecting means for detecting a vanishing point that is an intersection of two boundary areas detected by the boundary area detecting means;
The vanishing point detecting unit detects a boundary region of the image based on a vanishing point detected in an image captured a predetermined time before the time when the image was captured by the vanishing point detecting unit. A featured image processing apparatus.
An image processing method executed by an image processing apparatus that processes a plurality of images obtained by imaging a plurality of boundaries,
A boundary region detection step of detecting a boundary region captured in the image;
A parallax calculating step of calculating parallax of small areas constituting a plurality of the images;
A frequency calculation step of calculating the parallax frequency of the small area for each horizontal position of the image with respect to the parallax calculated by the parallax calculation step;
An inter-boundary small area calculating step of calculating an inter-boundary small area that is a predetermined small area between two boundary areas detected by the boundary area detecting step;
The running area detecting step of detecting the running area based on the frequency from the small area between the boundaries toward the small area at the horizontal end in the image;
An image processing method comprising: An image processing program for causing an image processing apparatus to execute processing of a plurality of images obtained by imaging a plurality of boundaries,
A boundary region detection step of detecting a boundary region captured in the image;
A parallax calculating step of calculating parallax of small areas constituting a plurality of the images;
A frequency calculation step of calculating the parallax frequency of the small area for each horizontal position of the image with respect to the parallax calculated by the parallax calculation step;
An inter-boundary small area calculating step of calculating an inter-boundary small area that is a predetermined small area between two boundary areas detected by the boundary area detecting step;
The running area detecting step of detecting the running area based on the frequency from the small area between the boundaries toward the small area at the horizontal end in the image;
An image processing program comprising: An imaging device for capturing an image;
An image processing apparatus that processes a plurality of images obtained by imaging a plurality of boundaries,
Boundary region detection means for detecting a boundary region captured in the image;
Parallax calculating means for calculating parallax of small areas constituting the plurality of images;
A frequency calculation means for calculating the parallax frequency of the small area for each horizontal position of the image with respect to the parallax calculated by the parallax calculation means;
An inter-boundary small area calculating means for calculating an inter-boundary small area that is a predetermined small area between two boundary areas detected by the boundary area detecting means;
The running area detecting means for detecting the running area based on the frequency from the small area between the boundaries toward the small area at the end in the horizontal direction in the image;
An image processing apparatus having
An image processing system comprising:   A moving apparatus comprising the image processing apparatus according to claim 1 or the image processing system according to claim 10.

Images