JP6313081B2 - In-vehicle image processing apparatus and vehicle system using the same - Google Patents

Description

  The present invention relates to an in-vehicle image processing apparatus for an automobile that detects an object based on information from an image sensor such as a camera, and performs an alarm or vehicle behavior control according to the detection result, and a vehicle using the same About the system.

  In recent years, a system for supporting a driver by controlling a vehicle using a sensor such as a camera or a sonar has been developed. For example, a system that detects lanes and alerts the driver by warning when the vehicle is likely to deviate, and prevents deviation by controlling the brakes and steering when the situation becomes unavoidable In addition, an autonomous parking system that detects a parking frame and automatically performs part or all of a driver's parking operation has been put into practical use.

  When a white line constituting a parking frame or a lane is detected using a camera, a shadow of a guard rail or a shadow of an electric wire is reflected on the road surface and appears to be as wide as the white line. As a countermeasure against such erroneous detection, for example, in Patent Document 1, when a lane is detected by a camera mounted on a vehicle, a road surface brightness histogram is created by setting a reference region and a road surface brightness. A technique for detecting a white line that deviates from the distribution is described.

JP 2006-268199 A

  However, for example, when sunlight is shining from behind the vehicle, most of the reference area is a road surface shaded by the vehicle, and erroneous detection due to the shadow cannot be removed. In addition, when the road is raining, the road surface gets wet and a distant scene is reflected, so the brightness of the road surface cannot be accurately captured.

  The present invention has been made in view of the above points, and the object of the present invention is to detect an object based on an optimum reference according to the road surface condition when detecting an object such as a white line. It is providing the vehicle-mounted image processing apparatus which implements, and a vehicle system using the same.

Vehicle image processing apparatus of the present invention for solving the aforementioned problems is a vehicle image processing apparatus that detects an object on the road surface, an image acquisition unit that acquires an image by imaging the vehicle periphery, of the image A road surface state estimation unit that estimates a road surface state around the vehicle based on a luminance distribution in the vicinity of the vehicle, a determination method setting unit that sets a determination method of an object according to the road surface state, and the target A target candidate detection unit that detects a target candidate that is estimated as a target, and a target determination unit that determines whether the target candidate is the target using the determination method. It is said.

  According to the present invention, the road surface state is estimated, and the object is determined using the determination method set according to the estimated road surface state, so that a part of the road surface is erroneously detected as the object. Therefore, it is possible to provide a vehicle system that performs highly reliable control based on this. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

The block diagram of the vehicle-mounted image processing apparatus in 1st Embodiment. Explanatory drawing of the process of the image acquisition part in 1st Embodiment. Explanatory drawing of the own vehicle vicinity area | region in 1st Embodiment. The figure which shows an example of a uniform state and its luminance histogram typically. The figure which shows typically an example of a shade sunlight state and its luminance histogram. The figure which shows typically an example of another brightness-and-dark separation state and its luminance histogram. The figure which shows an example of a reflection state and its luminance histogram typically. The figure which shows typically an example of a complicated state and its brightness | luminance histogram. The flowchart showing the process of the road surface state estimation part in 1st Embodiment. The figure which shows the difference | error of the brightness | luminance histogram of a uniform state, single Gaussian distribution, and both. The figure which shows the difference | error of the brightness | luminance histogram and single Gaussian distribution of a shade sunlight state. The figure which shows the difference | error of the brightness | luminance histogram of a reflective state, single Gaussian distribution, and both. The figure which shows the difference | error of the luminance histogram of a complicated state, single Gaussian distribution, and both. The figure which shows the difference | error of the brightness | luminance histogram of a uniform state, mixed Gaussian distribution, and both. The figure which shows the difference | error of the brightness | luminance histogram and mixed Gaussian distribution of a shade sunlight state. The figure which shows the difference | error of the brightness | luminance histogram of a reflection state, mixed Gaussian distribution, and both. The figure which shows the difference | error of the brightness | luminance histogram of a complicated state, mixed Gaussian distribution, and both. Explanatory drawing of the process of the road surface state estimation part in 1st Embodiment. The flowchart showing the process of the white line candidate detection part in 1st Embodiment. The flowchart showing a part of process of the white line determination part in 1st Embodiment. The figure which shows the brightness | luminance histogram of a white line, mixing Gaussian distribution, and the error of both. The figure which shows the difference | error of the brightness | luminance histogram of a sunshine in a shade, mixed Gaussian distribution, and both. The flowchart showing a part of process of the white line determination part in 1st Embodiment. The figure explaining whether a white line candidate is a radial form in a bird's-eye view image. The flowchart showing a part of process of the white line determination part in 1st Embodiment. The flowchart showing the process of the parking frame recognition part in 1st Embodiment. Explanatory drawing of the process of the parking frame recognition part in 1st Embodiment. The block diagram of another form of 1st Embodiment. The flowchart showing the process of the parking assistance control part in 1st Embodiment. The block diagram of the vehicle-mounted image processing apparatus in 2nd Embodiment. The flowchart showing the process of the stain | pollution | contamination detection part in 2nd Embodiment. The flowchart showing a part of process of the white line determination part in 2nd Embodiment. The flowchart showing a part of process of the white line determination part in 2nd Embodiment. The flowchart showing a part of process of the white line determination part in 2nd Embodiment. The flowchart showing the process of the parking frame recognition part in 2nd Embodiment. The flowchart showing the process of the parking assistance control part in 2nd Embodiment. The block diagram of the vehicle-mounted image processing apparatus in 3rd Embodiment. The flowchart showing a part of process of the obstacle determination part in 3rd Embodiment. The flowchart showing a part of process of the obstacle determination part in 3rd Embodiment. Explanatory drawing of the process of the obstruction determination part in 3rd Embodiment. The flowchart showing the process of the vehicle control part in 3rd Embodiment.

<First embodiment>
Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram of an in-vehicle image processing apparatus 1000 according to the first embodiment.
The in-vehicle image processing apparatus 1000 is incorporated in a camera device mounted on an automobile, an integrated controller, or the like, and is used for detecting an object from images captured by the cameras 1001 to 1004 of the camera device. In the present embodiment, a parking frame drawn by a white line on the road surface around the host vehicle is detected. The plurality of cameras 1001 to 1004 are attached at positions where the entire periphery of the vehicle can be imaged so as to include at least the road surface. In the present embodiment, the front part, the left part, the right part, Each is attached to the rear. Since the number of cameras and the mounting position need to be able to detect the shaded sun around the vehicle, it is sufficient that at least two cameras are mounted so as to capture images in directions away from each other. .

  The in-vehicle image processing apparatus 1000 is configured by a computer having a CPU, a memory, an I / O, and the like. A predetermined process is programmed and the process is repeatedly executed at a predetermined cycle T. As shown in FIG. 1, the in-vehicle image processing apparatus 1000 includes an image acquisition unit 1021, a road surface state estimation unit 1031, a determination method setting unit 1041, a white line candidate detection unit (object candidate detection unit) 1051, and a white line A determination unit (object determination unit) 1061 and a parking frame recognition unit 1071 are included.

  As shown in FIG. 2, the image acquisition unit 1021 acquires images 1011 to 1014 obtained by photographing the surroundings of the host vehicle from a plurality of cameras 1001 to 1004, and looks down from a virtual viewpoint in the sky by geometric conversion and synthesis. An overhead image 1015 is generated and stored on the RAM. It is assumed that geometric transformation / synthesis parameters are set in advance by calibration performed at the time of vehicle shipment. Note that the bird's-eye view image 1015 is a two-dimensional array and is represented by IMGSRC [x] [y]. x and y indicate the coordinates of the image, respectively.

  The road surface state estimation unit 1031 acquires the luminance histogram HST [b] from the own vehicle vicinity region 1032 in the bird's-eye view image 1015, and estimates whether the road surface state around the own vehicle belongs to a plurality of preset road surface states. To do. The own vehicle vicinity area 1032 is an area set in the vicinity of the own vehicle in the bird's-eye view image 1015, and it can be considered that the road surface around the own vehicle is always captured in the own vehicle vicinity area 1032. The luminance histogram HST [b] is a one-dimensional array, and b represents a histogram bin. In the present embodiment, it is estimated which of the four types of road surface conditions, ie, [uniform state], [bright / dark separation state], [shade-sunshine state], and [complex state].

  The determination method setting unit 1041 sets a determination method for determining a white line according to the road surface state estimated by the road surface state estimation unit 1031. In the present embodiment, when the road surface state is [uniform state], [bright / dark separation state], or [shade-sunshine state], a determination is made using a luminance range obtained from the luminance histogram or a Gaussian model of the luminance histogram. When the road surface state is [complex state], a determination method using an image pattern is set.

  The white line candidate detection unit 1051 detects a white line candidate LC [n] from the overhead image 1015. The white line candidate LC [n] is a one-dimensional array whose elements are tables having information such as the coordinates of white line candidates, and n represents an ID when a plurality of white line candidates are detected.

  The white line determination unit 1061 determines whether the white line candidate LC [n] is a white line or a road surface based on the determination method set by the determination method setting unit 1041, and determines the white line LN [m]. The white line LN [m] is a one-dimensional array having a table having information such as the coordinates of the white line as an element, and m represents an ID when a plurality is detected.

  The parking frame recognition unit 1071 detects the parking frame PS [k] based on the white line LN [m]. The parking frame PS [k] is a one-dimensional array having a table having information such as coordinates of the parking frame as elements, and k represents an ID when a plurality of detections are detected.

[Road surface state estimation unit]
Next, the contents of processing in the road surface state estimation unit 1031 will be described with reference to FIGS. 3-1 to 3-6, FIGS.

  FIG. 4A is a flowchart showing a flow of processing of the road surface state estimation unit 1031. Hereinafter, details of the processing will be described. First, in step S401, a luminance histogram is created from the vehicle vicinity area.

  FIG. 3A is an explanatory diagram of the vehicle vicinity area 1032, and in FIGS. 3A to 3C, the upper side of the drawing indicates the front side of the vehicle. Here, when the own vehicle vicinity area 1032 is fixed as shown in FIG. 3-1 (a), and as shown in FIGS. 3-1 (b) and (c), Both cases where the vehicle vicinity area 1032 is variable will be described.

  As shown in FIG. 3A, when the own vehicle vicinity region 1032 is fixed, it is set near the center in the overhead image 1015. Since the black part shown inside the own vehicle vicinity area 1032 is a part which cannot be imaged with the cameras 1001 to 1004, this area is excluded from the subject of the own vehicle vicinity area 1032. As shown in FIGS. 3-1 (b) and (c), when the own vehicle vicinity region 1032 is made variable according to the own vehicle behavior, shift position information of the transmission of the vehicle is acquired and based on the information. When the host vehicle is moving forward, the rear side of the host vehicle is set larger as shown in FIG. 3B, and when the host vehicle is moving backward, the front side of the host vehicle is set larger as shown in FIG. To do. Similarly, the central portion is excluded.

  FIG. 3-1 (d) is an example of the brightness histogram HST [b] acquired from the own vehicle vicinity area 1032. The luminance histogram HST [b] represents the frequency of the luminance of the image, and is represented by a one-dimensional graph with the horizontal axis representing luminance and the vertical axis representing frequency.

  FIGS. 3-2 to 3-6 are diagrams schematically illustrating examples of each road surface state and its luminance histogram. Fig. 3-2 is a diagram schematically showing an example of a uniform state and its luminance histogram, Fig. 3-3 is a diagram schematically showing an example of a shaded and sunny state and its luminance histogram, and Fig. 3-4 is the others. FIG. 3-5 schematically illustrates an example of a reflection state and its luminance histogram, and FIG. 3-6 schematically illustrates an example of a complex state and its luminance histogram. FIG.

  The road surface of the bird's-eye view image 1015 shown in FIG. 3-2 is in a uniform state that has no shade and shadow and can be regarded as having almost a single luminance. As described above, the luminance histogram when the road surface state is a uniform state has a large distribution of the road surface region, and is unimodal with one large peak.

  The road surface of the bird's-eye view image 1015 shown in FIG. 3-3 (a) is in a shaded and sunny state, which is a kind of light / dark separation state in which the shadow of the vehicle or building is reflected in the sun and the brightness of the road is divided into light and dark. The road surface of the bird's-eye view image 1015 shown in FIG. 3-4 (a) is another light / dark separation state in which the brightness of the road surface is divided into light and dark due to road construction or the like. The road surface of the bird's-eye view image 1015 shown in FIG. 3-5 (a) is a reflection state in which the structure reflected on the road surface wet by rain or the like is reflected. The road surface of the bird's-eye view image 1015 shown in FIG. 3-6 (a) is in a complicated state in which, for example, light and shadows from a plurality of directions are reflected, and it is difficult to approximate the luminance histogram by a preset Gaussian distribution. is there.

  Among them, the luminance histograms shown in FIGS. 3-3 (b), 3-4 (b), and 3-5 (b) are different in shape from each other in the shaded sunlight state, the other light / dark separation state, and the reflection state. Is generally bimodal with two large peaks. Then, three or more peaks appear in the luminance histogram in the complicated state shown in FIG. 3-6 (b).

  Next, in step S402, the luminance histogram is analyzed. In the analysis, the luminance histogram HST [b] is approximated by a single Gaussian distribution having one peak or a mixed Gaussian distribution having two peaks, and the error is obtained. Hereinafter, the weights α_ {1, 2,..., G}, the average μ_ {1, 2,..., G}, and the variance σ_ {1, 2 are the parameters of the mixed Gaussian distribution expressed by the distribution number G. ,..., G} are expressed as a table GPTBL (G). An EM algorithm is generally used as a method of approximating a certain distribution by the mixed Gaussian distribution GPTBL (G). Since this method is publicly known, the description is omitted here.

  In the case of approximation by the EM algorithm, it is necessary to specify the number G of the Gaussian distribution. Therefore, in this embodiment, G = 1 and 2 are used, and the approximation error of G = 1 is Err (1) and G = 2. Let the approximation error be Err (2). The error between the luminance histogram HST [b] and the Gaussian distribution GPTBL (G) is calculated by the following equation.

Err (G) = Σ (b = 0 to 255): MIN (HSTNML [b], HSTMLD (G) [b]) / MAX (HSTNML [b], HSTMLD (G) [b])
Here, HSTNML [b] is obtained by normalizing the luminance histogram HST [b] so that the total value thereof is 1. HSTMLD (G) [b] is a distribution obtained from the table GPTBL (G), and is normalized so that the total value thereof is also 1.

  Examples in the case of approximating G = 1 with respect to the examples of FIGS. 3-2 to 3-5 are shown in FIGS. FIGS. 4-2 (a) to (c) are examples when the road surface in the uniform state shown in FIG. 3-2 is approximated by G = 1. For the luminance histogram in FIG. 4A, the distribution represented by the weight α_1, the average μ_1, and the variance σ_1 of the G = 1 Gaussian distribution obtained by the EM algorithm is shown in FIG. The error Err (1) in FIGS. 4-2 (a) and (b) is the area of the region shown in black in FIG. 4-2 (c).

  Similarly, FIGS. 4-3 (a) to (c) are examples in the case of approximating G = 1 with respect to the road surface in the shaded and sunny state shown in FIG. 3-3. The Gaussian distribution calculated from the luminance histogram of FIG. 4-3 (a) becomes FIG. 4-3 (b), and the error Err (1) of FIGS. 4-3 (a) and (b) is shown in FIG. 4-3 (c). ) Is the area of the region shown in black. Similarly, FIGS. 4-4 (a) to (c) are examples applied to the road surface in the reflective state shown in FIG. 3-5, and FIGS. 4-5 (a) to (c) are similar to FIG. 6 is an example applied to a road surface in a complicated state shown in FIG.

  Next, examples in the case of approximating G = 2 with respect to the examples of FIGS. 3-2 to 3-5 are shown in FIGS. 4-6 to 4-9. 4-6 (a) to (c) are examples in the case of approximating G = 2 with respect to the uniform road surface shown in FIG. 3-2. With respect to the luminance histogram of FIG. 4-6 (a), distributions represented by weights α_1, α_2, average μ_1, μ_2, variances σ_1, and σ_2 of the G = 2 Gaussian distribution obtained by the EM algorithm are shown in FIG. This is indicated by a thick line in b). The error Err (2) in FIGS. 4-6 (a) and (b) is the area of the region shown in black in FIG. 4-6 (c).

  Similarly, FIGS. 4-7 (a) to (c) are examples in the case of approximating G = 2 with respect to the road surface in the shaded and sunny state shown in FIG. 3-3. The Gaussian distribution calculated from the luminance histogram of FIG. 4-7 (a) becomes FIG. 4-7 (b), and the error Err (1) of FIGS. 4-7 (a) and (b) is shown in FIG. ) Is the area of the region shown in black. Similarly, FIGS. 4-8 (a) to (c) are examples applied to the reflective road surface shown in FIG. 3-5, and FIGS. 4-9 (a) to (c) are similar to FIGS. 6 is an example applied to a road surface in a complicated state shown in FIG.

  As shown above, a unimodal histogram such as a uniform road surface has a smaller error with respect to the distribution of G = 1. On the other hand, the bimodal histogram, like the road surface in the shaded and sunny state, has a small error with respect to the distribution of G = 2. Furthermore, a histogram that does not correspond to either of them, such as a road surface in a complicated state, has a large error with respect to any approximation.

  In addition, when the distribution of G = 2 is applied to a unimodal histogram like a uniform road surface, the error tends to be smaller than the error of G = 1. Accordingly, in the following determination, the approximation error at G = 1 is determined first, and the approximation error at G = 2 is determined only when the error is large.

Since there are various methods for comparing the two histograms, another method may be used for the calculation of the error Err (G). For example, when the Bhattacharya distance is applied, the calculation is performed using the following equation.
Err (G) = 1−Σ (b = 0 to 255): SQRT (HSTNML [b] · HSTMLD (G) [b])

  Next, in step S403, it is determined whether the luminance histogram HST [b] can be approximated by a single Gaussian distribution GP (1). Here, if the error Err (1) of GP (1) is equal to or smaller than the threshold value TH_Err1, it is determined that approximation is possible, and if it is larger than the threshold value, it is determined that approximation is impossible and the process moves to step S404.

  Next, in step S404, it is determined whether the luminance histogram HST [b] can be approximated by the mixed Gaussian distribution GP (2). Here, if the error Err (2) in the case of GP (2) is less than or equal to the threshold value TH_Err2, it is determined that approximation is possible, and the process moves to step S406 to analyze the road surface pattern. It judges with having carried out, and moves to Step S409.

  In the approximation in step S402, approximation with G = 3, G = 4, which is larger than G = 2, may be attempted. In this case, in step S404, the smallest approximation error in G = 2, 3, and 4 is used. An approximation error is selected, and whether or not approximation is possible is determined based on whether or not the value is equal to or less than a threshold value.

  When it is determined in step S403 that the luminance histogram HST [b] can be approximated by a single Gaussian distribution GP (1) (YES in step S403), the road surface state is estimated as [uniform state] in step S405. The road surface state is “uniform state” as shown in FIG. 3-2, which means that there is no shade or shadow on the road surface and can be regarded as almost a single luminance. Even when a white line such as a parking frame is present on the road surface, the number of pixels on the road surface is overwhelmingly larger than the number of pixels on the white line in the image, so that it can be approximated to a single Gaussian distribution.

  If it is determined in step S404 that the luminance histogram HST [b] can be approximated by the mixed Gaussian distribution GP (2) (YES in step S404), it is estimated that the road surface is separated into light and dark [light / dark separation state]. Then, in order to analyze the state in more detail, the processing after step S406 is executed. In step S406, road surface pattern analysis is performed. In the road surface pattern analysis, processing for recognizing the light and dark shape of the road surface is performed. Then, based on the bright and dark shape of the road surface recognized in step S406, it is determined in step S407 whether or not [brightness / darkness separation state] is [reflection state].

  If it is determined in step S407 that it is not [reflective state] (NO), the process proceeds to step S408, and it is estimated that the road surface state is either [shade-sunshine state] or [other light / dark separation state]. To do. On the other hand, if it is determined in step S407 that the state is [reflection state] (YES), the process proceeds to step S409, and the road surface state is estimated to be [complex state (including [reflection state]).

  FIG. 5 is a diagram showing a specific example of the road surface state, FIG. 5 (a) shows a shaded and sunny state, FIG. 5 (b) shows other light / dark separation states, and FIG. 5 (c) shows a reflection state. Show. The road surface state being “shade and sunny state” refers to a state in which the shadow of the vehicle is reflected in the image. And [other light-and-dark separation state] means that, for example, as shown in FIG. 5 (b), the road surface is reflected by a state in which a shadow of an object other than the own vehicle such as a building or a wall is reflected, or road construction. The state in which the brightness of the is partially different. The “reflection state” means a state in which a structure reflected on a road surface wet by rain or the like is reflected as shown in FIG. 5C, for example.

  The road surface pattern analysis is performed by analyzing the overhead image 1015 using the approximate mixed Gaussian distribution parameter GP (2). Hereinafter, of the two Gaussian distributions of GP (2), a distribution with a large average luminance is GPH, and a distribution with a low average luminance is GPL.

  First, for each pixel of the overhead image 1015, a pixel belonging to GPL is extracted from two Gaussian distributions. Whether the pixel belongs or not can be determined by substituting the luminance of each pixel into GPL and GPH, and whether or not the output value of GPL is larger than the output value of GPH.

  Next, a binary image is generated by extracting only pixels belonging to the GPL in the overhead image 1015, and the shape is analyzed. For example, as shown in FIG. 5A, when a pixel belonging to a Gaussian distribution with a small average luminance extends in a certain direction from within the vicinity of the host vehicle, the road surface state is [shade-sunshine state]. judge. When it is determined that the road surface state is [shade / sunshine state], the GPH pixel area Sa corresponds to the sun road surface, and the GPL pixel area Sb corresponds to the shade road surface.

  For example, as shown in FIG. 5 (b), when the area near the vehicle is divided into the GPL pixel area Sb and the GPH pixel area Sa, for example, the pixels belonging to the GPL are present in half of the image. Then, it is determined that the road surface state is [other light / dark separation state]. When the GPL pixel region Sb has a complicated shape due to reflection of a structure reflected on the road surface wetted by rain or the like as shown in FIG. (YES in step S407). The above determination can be made by performing a labeling process on a binary image obtained by extracting pixels belonging to the GPL and viewing the shape of each label.

  In step S407, if the road surface state is not “reflection state”, the process moves to step S408. If the road surface state is “reflection state”, the process moves to step S409. In step S408, it is determined that the luminance histogram can be approximated by a Gaussian distribution set in advance, and the road surface state is estimated to be [shade sunshine state] or [other light / dark separation state]. In step S409, it is determined that it is difficult to approximate the luminance histogram by a preset Gaussian distribution or the shape is complicated, and the road surface state is estimated to be [complex state]. [Reflection state] is treated as a subset of [Complex state].

  As described above, the road surface state estimation unit 1031 estimates which model is the most applicable to the unimodality, bimodality, and other models of the own vehicle vicinity region, and further estimates the parameters in the fitted model.

[Judgment method setting section]
Next, the contents of processing of the determination method setting unit 1041 will be described.
In the present embodiment, two types of determination methods are mainly used in cases where the road surface state is estimated as [complex state] and in other cases. If the road surface state is determined to be other than [complex state], determination using the mixed Gaussian distribution model GP (G) used in the road surface state estimation unit 1031 is performed. On the other hand, when the road surface state is determined as [complex state], determination using a luminance pattern is performed. Details of the luminance pattern will be described in the white line determination unit 1061.

[White line candidate detection unit]
Next, the contents of processing in the white line candidate detection unit 1051 will be described with reference to FIG.
FIG. 6 is a flowchart showing the processing flow of the white line candidate detection unit 1051. The white line candidate detection unit 1051 may be performed on the entire overhead image 1015 or may limit the processing region. In this embodiment, based on the shift position, the upper half of the overhead image 1015 is set as the processing area when the host vehicle is stopped and moving forward, and the lower half is set when the host vehicle is moving backward.

  First, in step S601, a vertical edge filter is applied to each line in the processing region while scanning from the left side to the right side of the image. Hereinafter, steps S602 and S603 are performed for each line. In step S602, an edge point where the output value of the edge filter reaches a peak is detected. The peak is detected at the rising edge (change point of luminance from dark to light) Eu [nu] falling edge (change point of luminance from light to dark) Ed [nd].

  Since the white line on the road surface has a higher luminance value than the road surface, a rising edge exists on the left side of the white line and a falling edge exists on the right side. In order to capture the feature, in step S603, among the rising edge point Eu [nu] and the falling edge point Ed [nd] detected in step S602, a predetermined range on the right side of the image from the rising edge (predefined and detected) Only the edge pair Ep [np] in which the falling edge exists within the maximum thickness of the white line is left, and the other single edges are deleted. The above S601 to S603 are executed for each line in the processing area.

  Next, in step S604, the edge pairs Ep [np] arranged in a straight line are grouped to generate a straight line candidate group Lg [ng]. Here, edge pairs that are not linear are removed. The grouping of white lines arranged on a straight line is possible by using a known Hough transform. Next, in step S605, lines having a predetermined length or less are removed from the lines of the straight line candidate group Lg [ng]. In step S606, the coordinates of the upper and lower edges of the rising and falling edges of the remaining group are stored as elements of the white line candidate LC [n].

[White line judgment part]
Next, the content of processing in the white line determination unit 1061 will be described with reference to FIGS. 7-1 to 7-3, FIGS. 8-1, 8-2, and 9. FIG. 7A is a flowchart illustrating a processing flow of the white line determination unit 1061.

  First, in step S701, it is confirmed whether or not the road surface state determined by the road surface state estimation unit 1031 is a [complex state]. If it is not [complex state], the process proceeds to step S7011, and if it is [complex state], the process proceeds to step S706. In step S7011, it is confirmed whether or not the road surface state determined by the road surface state estimation unit 1031 is [shade and sunny state]. If it is not [shade-sunshine state], the process proceeds to step S702. If it is [shade-sunshine state], the process proceeds to step S707. Hereinafter, the processing from step S702 to step S705 is repeated for the number of white line candidates LC [n].

  First, in step S702, a luminance histogram HSTLC [n] inside the white line candidate LC [n] is created. Next, in step S703, it is determined whether the white line candidate luminance (inner luminance histogram of the white line candidate) is included in the road surface luminance reference. Here, an error Err_LNGP between the luminance histogram HSTLC [n] inside the white line candidate LC [n] and the mixed Gaussian distribution model GP (G) set by the determination method setting unit 1041 is calculated and compared with the threshold TH_LNGP. To do. If the error Err_LNGP is larger than the threshold value TH_LNGP, it is determined that the brightness of the white line candidate is not included in the road surface brightness reference (NO in step S703), and the process moves to step S704. It determines with it being contained in a brightness | luminance reference | standard (it is YES at step S703), and moves to step S705.

  In step S704, the white line candidate is determined to be a white line, and the information of the white line candidate LC [n] is copied to the white line LN [m]. On the other hand, in step S705, it is determined that the white line candidate is not a white line, and the information of the white line candidate LC [n] is not copied to the white line LN [m] but is erased. In step S706, the process when it is determined that the state is [complex state] described below is entered.

  A specific example of the above-described white line determination process will be described with reference to FIGS. 7-2 and 7-3. FIG. 7-2 (a) is a scene in which a parking frame line in front of the host vehicle is detected as a white line candidate LC. Here, the processing for the right white line candidate LC among the left and right white line candidates will be described (the same applies to the processing for the left white line candidate).

  FIG. 7-2 (b) shows the luminance histogram HSTLC [n] of the inner region of the white line candidate LC [n] (inside the dotted rectangle on the right side of FIG. 7-2 (a)), and FIG. The histogram of the mixed Gaussian distribution model GP (G) set by the method setting unit 1041, FIG. 7-2 (d) displays the error of the histograms of FIGS. 7-2 (b) and (c). .

  The luminance histogram shown in FIG. 7B is obtained as the luminance histogram HSTLC [n] of the inner region of the white line candidate LC [n], and the mixed Gaussian distribution model GP (G) set by the determination method setting unit 1041 is obtained. When the luminance histogram is as shown in FIG. 7-2 (c), the histograms of FIGS. 7-2 (b) and (c) are compared with each other as shown in FIG. 7-2 (d). The position of is very different. Therefore, when the error of these two histograms is obtained using the method described in the road surface state estimation unit 1031, a large value exceeding the threshold value TH_LNGP is obtained and determined as a white line.

  On the other hand, the example of FIG. FIG. 7-3 (a) is a scene in which a shadow (GPL pixel area Sb) exists in front of the host vehicle, and a sunlit part (GPH pixel area Sa) between the shadows is detected as a white line candidate LC. Here, the processing for the right white line candidate LC among the left and right white line candidates will be described (the same applies to the processing for the left white line candidate).

  FIG. 7-3 (b) shows the luminance histogram HSTLC [n] of the inner region of the white line candidate LC [n] (inside the dotted rectangle on the right side of FIG. 7-3 (a)), and FIG. The histogram of the mixed Gaussian distribution model GP (G) set by the method setting unit 1041 and FIG. 7-3 (d) display the errors of the histograms of FIGS. 7-3 (b) and (c). .

  The luminance histogram shown in FIG. 7-3 (b) is obtained as the luminance histogram HSTLC [n] of the inner region of the white line candidate LC [n], and the mixed Gaussian distribution model GP (G) set by the determination method setting unit 1041 is obtained. When the luminance histogram is as shown in FIG. 7-3 (c), comparing the histograms of FIGS. 7-3 (b) and (c), as shown in FIG. The position is almost the same. Accordingly, when the error between these two histograms is obtained using the method described in the road surface state estimation unit 1031, a small value can be obtained. Therefore, the value is equal to or less than the threshold value TH_LNGP, and it is determined that the line is not a white line and is erased.

  In step S706, the process described in FIG. 8-1 is executed. FIG. 8A is a process when it is determined that the road surface is in the “complex state”. The processing from step S801 to step S804 is repeated for the number of white line candidates LC [n]. First, in step S <b> 801, a region including a white line candidate is set for a luminance pattern around the white line candidate, that is, the overhead image 1015. Next, in step S802, it is determined whether the luminance pattern of the white line candidate is radial, that is, whether the shape of the region set to include the white line candidate in the overhead image 1015 is radial. As described with reference to FIG. 2, the bird's-eye view image 1015 is a virtual viewpoint image from the sky.

  If there is road surface reflection or the like in which the structure is reflected and reflected on the road surface wet by rain or the like, a pattern as shown in FIG. 5C appears on the road surface. Since this pattern extends radially from the camera installation position on the bird's-eye view image 1015, the white line candidate is determined by determining whether or not the image in the region set to include the white line candidate is radial from the camera installation position. You can tell whether it is due to road surface reflection or true white line.

  In the present embodiment, the following method is used as a method for determining whether the beam is radial. First, an edge angle DE [x] [y] is calculated from each image in an area set to include white line candidates. Next, an angle DC [x] [y] formed by the position of each pixel and the camera installation position is calculated. Then, the difference between DE [x] [y] and DC [x] [y] is obtained, and it is determined whether or not the value is within the threshold TH_DEDC. The above processing is performed for all the pixels in the region set to include white line candidates, and the number of pixels that are within the threshold is counted. Then, a value RR obtained by dividing the counted number of pixels by the area of the region is obtained. If the value is equal to or greater than the threshold value TH_RR, it is determined to be radial, otherwise it is determined not to be radial. Here, threshold values TH_DEDC and TH_RR are set in advance.

  If it is determined in step S802 that the brightness pattern of the white line candidate is not radial (NO in step S802), the process moves to step S803. In step S803, the white line candidate is determined as a white line, and the information of the white line candidate LC [n] is copied to the white line LN [m]. On the other hand, if it is determined in step S802 that the brightness pattern of the white line candidate is radial (YES in step S802), the process moves to step S804. In step S804, it is determined that the white line candidate is not a white line, and the information of the white line candidate LC [n] is not copied to the white line LN [m] but is erased.

  An example of radial white line candidates and an example of non-radial white line candidates in the above processing will be described with reference to FIG. FIG. 8-2 is a diagram for explaining whether or not white line candidates are radial in the overhead image. FIG. 8-2 (a) shows the installation positions of the cameras 1001 to 1004 on the overhead image 1015. Reflection on the road surface is a direction extending radially from the camera position.

  FIG. 8B is an example of a white line candidate LC that is not radial. In this case, since the edge angle of the white line candidate LC is not radial, the position of the camera 1001 is not passed. That is, the camera 1001 is not located on the extension line of the white line candidate LC. Therefore, the error between the edge angle DE [x] [y] of each pixel and the angle DC [x] [y] formed by each pixel and the camera installation position becomes large.

  FIG. 8-2 (c) is an example of a radial white line candidate LC. In this case, since the angle of the edge of the white line candidate LC is radial, it passes through the position of the camera 1001. That is, the camera 1001 is positioned on the extension line of the white line candidate LC. Therefore, the error between the edge angle DE [x] [y] of each pixel and the angle DC [x] [y] formed by each pixel and the camera installation position becomes small.

  Furthermore, a process when it is determined in FIG. 9 that the road surface state is [shade-sunshine state] will be described. If it is determined in step S7011 that the road surface state is [shade and sunny], the process moves to step S707, and the process described in FIG. 9 is executed. FIG. 9 shows a process when it is determined that the road surface is [shade and sunny].

  Hereinafter, the processing from step S901 to step S906 is repeated by the number of white line candidates LC [n].

  In step S901, the brightness histogram HSTLC [n] inside the white line candidate LC [n] is acquired. Next, in step S902, a luminance histogram HSTLCEX [n] outside the white line candidate LC [n] is acquired. Next, in step S903, an error Err_LNGPH between the luminance histogram HSTLC [n] inside the white line candidate LC [n] and the sunny road surface distribution model GPH is calculated and compared with a threshold value TH_LNGPH. If the error Err_LNGPH is greater than the threshold TH_LNGPH, it is determined that it is not included in the sun road surface brightness reference (NO in step S903), and the process proceeds to step S906. YES, and the process proceeds to step S904.

  Next, in step S904, an error Err_LNGPL between the luminance histogram HSTLCEX [n] outside the white line candidate LC [n] and the shaded road surface distribution model GPL is calculated and compared with a threshold TH_LNGPL. If the error Err_LNGPL is larger than the threshold value TH_LNGPL, it is determined that it is not included in the shaded road surface brightness reference (NO in step S904), and the process moves to step S906. It determines with (it is YES at step S904), and moves to step S905.

  In step S906, the white line candidate is determined to be a white line, and the information of the white line candidate LC [n] is copied to the white line LN [m]. In step S905, it is determined that the white line candidate is not a white line, and the information on the white line candidate LC [n] is not copied to the white line LN [m] but is erased.

  As described above, the white line determination unit 1061 has described the process of whether to include the white line candidate LC [n] in the white line LN [n] based on the determination method according to the brightness of the white line candidate LC [n] and the road surface state. . In the present embodiment, when the road surface state is a state other than [complex state], the error histogram HSTLC [b] inside the white line candidate LC [n] and the mixed Gaussian distribution approximation model GP (G) are large in error. Judgment based on this. When the road surface state is [complex state], the determination is made based on the degree to which the luminance pattern of the region including the white line candidate LC [n] has a radial feature.

  Further, when the road surface state is [shade-sunshine state], the magnitude of error between the brightness histogram HSTLC [b] inside the white line candidate LC [n] and the Gaussian distribution model GPH of the sun road surface, and the white line The determination is based on the magnitude of the error between the brightness histogram HSTLCEX [b] outside the candidate LC [n] and the shaded Gaussian distribution model GPL. By such processing, determination according to the road surface condition is possible, and erroneous recognition can be prevented.

[Parking frame recognition unit]
Next, the contents of processing in the parking frame recognition unit 1071 will be described with reference to FIG.
FIG. 10 is a flowchart showing a processing flow of the parking frame recognition unit 1071.

  First, in step S101, two lines LN [mL] and LN [mR] are selected from the white line LN [m]. Next, in step S102, it is determined whether or not the angular difference θ in the extending direction of the two white lines LN [mL] and LN [mR] selected in step S101 is equal to or smaller than a predetermined value (Thθmax). That is, in step S102, it is determined whether or not the two white lines are substantially parallel. If a positive determination is made in step S102, the process proceeds to step S103, and it is determined whether or not the interval W between the two white lines LN [mL] and LN [mR] is within a predetermined range (ThWmin or more and ThWmax or less). That is, in step S103, it is determined whether or not two white lines are aligned at an interval that can be considered as two white lines constituting the parking frame.

  If an affirmative determination is made in step S103, the process proceeds to step S104, and it is determined whether or not the deviation ΔB between the lower ends of the two white lines is within a predetermined range (from ThBmin to ThBmax). Here, the shift of the lower ends of the two white lines will be described with reference to FIG. As shown to the left figure (a) of FIG. 11, it parks so that the vehicle 1 may be parked in parallel or at right angles with respect to the moving direction of the vehicle 1 on the site of the parking lot (vertical direction and horizontal direction on the paper surface of FIG. 11). When the frame 23 is provided and the parking frame line 23L is drawn so as to correspond to the parking frame 23, the shift ΔB between the lower ends of the two white lines on the paper surface is small.

  However, as shown in the right figure (b) of FIG. 11, depending on the parking lot, the vehicle 10 is not parallel or perpendicular to the moving direction of the vehicle 10 (vertical direction and horizontal direction on the paper surface of FIG. 11). A parking frame 23 may be provided so as to park, and a parking frame line 23 </ b> L may be drawn so as to correspond to the parking frame 23. In this case, as indicated by the distance B, the lower end position of the parking frame line 23L constituting the parking frame 23 is shifted. That is, the deviation between the lower ends of the two white lines is large. This is because the lower end position of the parking frame line 23L is shifted so that the parking frame line 23L on the right side of the drawing does not protrude from the area where the vehicle 1 travels below the page.

For example, in a highway service area or parking area, a line segment perpendicular to the extending direction of the parking frame line 23L from the lower end of the parking frame line 23L on the left side of the left figure (b) of FIG. It is determined that the crossing angle ρ with the line connecting the lower ends of 23L is 0 degree, 30 degrees, or 45 degrees. Moreover, the range of the value which can be taken also about the width W of the parking frame 23 is decided. Therefore, in the present embodiment, the value of the distance B that can be taken when the intersection angle ρ is one of 0 degree, 30 degrees, and 45 degrees is stored in advance in a memory (not shown) or the like.

  In step S104 of the present embodiment, the distance B calculated from the selected LN [mL] and LN [mR] is compared, and the difference is within the predetermined range described above (ThBmin or more and ThBmax or less). By judging whether or not, it is judged whether or not the two white lines are white lines (parking frame line 23L) constituting one parking frame 23.

  If an affirmative determination is made in step S104, the process proceeds to step S105, and the coordinates of the four corners of the rectangular parking frame 23 composed of two white lines LN [mL] and LN [mR] Register as information PS [k]. If step S105 is performed, it will progress to step S106 and it will confirm whether the process mentioned above about arbitrary two white lines was performed regarding all the white lines (parking frame line) based on the information output from the white line determination part 1061. . If an affirmative determination is made in step S106, the result obtained by the above-described process is output, and the process of the parking frame recognition unit 1071 is terminated. If a negative determination is made in step S106, the process returns to step S101.

  As described above, the white line determination method can be switched according to the road surface condition to suppress erroneous recognition, and the accuracy of parking frame recognition can be improved.

  In this embodiment, the monochrome camera has been described. However, color information (color information) may be used. When color information is used, it can be realized by performing the above processing for all three channels of R, G, and B.

  Furthermore, FIG. 12 is a block diagram in the case where the parking frame recognition result PS [k] detected in the present embodiment is transmitted to another control device mounted on the same vehicle, and the parking support control is performed. . Here, the parking support control unit 1081 acquires the position information of the parking frame recognition result PS [k], and performs the parking support control.

FIG. 13 is a flowchart showing a processing flow of the parking support control unit 1081.
First, in step S1301, position information of the parking frame recognition result PS [k] is acquired. The parking frame recognition result PS [k] may be acquired by directly inputting a signal of the in-vehicle image processing apparatus 1000 to the control apparatus, or acquired by performing communication using a LAN (Local Area Network). May be.

  Next, a parking locus is calculated in step S1302. The parking locus is generated based on the position of the parking frame recognition result PS [k]. Next, in step S1303, it is determined whether parking is possible. If the vehicle cannot be parked even if the vehicle is turned back several times, the process moves to step S1305 and parking assistance is stopped. On the other hand, if the vehicle can be parked, the process moves to step S1304, and at least one of steering, accelerator, and brake is controlled to assist parking in the target parking frame.

<Second embodiment>
Next, a second embodiment of the in-vehicle image processing apparatus of the present invention will be described below with reference to the drawings.

  FIG. 14 is a block diagram illustrating a configuration of the in-vehicle image processing apparatus 2000 according to the second embodiment. In the following description, only portions different from the in-vehicle image processing apparatus 1000 in the first embodiment described above will be described in detail, and the same portions will be denoted by the same reference numerals and detailed description thereof will be omitted.

  What is characteristic in the present embodiment is that it has a stain detection unit 2091 and performs processing of the white line determination unit 2061 using the information.

  The in-vehicle image processing device 2000 is incorporated in a camera device mounted on a car, an integrated controller, or the like, and detects an object from images captured by the cameras 1001 to 1004. In the form, it is configured to detect an obstacle ahead of the host vehicle.

  The in-vehicle image processing apparatus 2000 is configured by a computer having a CPU, a memory, an I / O, and the like, and a predetermined process is programmed and the process is repeatedly executed at a predetermined cycle.

  As illustrated in FIG. 14, the in-vehicle image processing apparatus 2000 includes an image acquisition unit 1021, a road surface state estimation unit 1031, a determination method setting unit 1041, a white line candidate detection unit 1051, a dirt detection unit 2091, and a white line determination. A unit 2061, a parking frame recognition unit 2071, and a parking support control unit 2081.

  As shown in FIG. 2, the image acquisition unit 1021 acquires images 1011 to 1014 obtained by photographing the surroundings of the vehicle from cameras 1001 to 1004 attached at positions where the surroundings of the vehicle can be imaged, and performs geometric transformation / combination. By doing so, a bird's-eye view image 1015 looking down from the sky virtual viewpoint is generated and stored on the RAM. It is assumed that geometric transformation / synthesis parameters are set in advance by calibration performed at the time of vehicle shipment.

  The road surface state estimation unit 1031 acquires the luminance histogram HST [b] from the own vehicle vicinity region 1032 in the overhead image 1015, and estimates the road surface state. In this embodiment, four types of road surface conditions are estimated: [uniform state], [bright / dark separation state], [shade-sunshine state], and [complex state].

  The determination method setting unit 1041 sets a determination method for determining a white line according to the road surface state estimated by the road surface state estimation unit 1031. In the present embodiment, in the case of [uniform state], [brightness / darkness separation state], and [shade-sunshine state], determination is performed using the luminance range obtained from the luminance histogram or the Gaussian model of the luminance histogram, and [complex state]. ], Determination using an image pattern is performed.

  The white line candidate detection unit 1051 detects a white line candidate LC [n] from the overhead image 1015. The dirt detection unit 2091 calculates the lens dirt position of the camera from the overhead image 1015. The white line determination unit 2061 sets the white line reliability LRLB [n] for the white line candidate LC [n] based on the determination criterion set by the determination method setting unit 1041 and the stain information detected by the stain detection unit 2091. calculate. The parking frame recognition unit 2071 detects the parking frame PS [k] based on the white line candidate LC [n] and the white line reliability LRLB [n].

  Furthermore, the parking assist control unit 2081 is mounted in a control device mounted on the same vehicle as the vehicle-mounted image processing device 2000, and the position of the parking frame recognition result PS [k] received from the vehicle-mounted image processing device 2000. Using the information and reliability, an optimal parking locus is calculated according to the positional relationship with the host vehicle, and the steering, the accelerator, and the brake are controlled, and the vehicle is parked in the target parking frame.

[Dirt detection unit]
The content of the process regarding an example of the stain | pollution | contamination detection part 2091 is demonstrated using FIG.
FIG. 15 is a flowchart showing the flow of processing of the dirt detection unit 2091 in this embodiment.

  In step S1401, the overhead image 1015 is divided into a plurality of small areas. It is assumed that how to divide the small area is set in advance. In step S1402, a histogram is created for each small region. In step S1403, the histogram created in step S1402 is compared with the histogram created when the process of FIG.

  In step S1404, each small region is evaluated based on the comparison performed in step S1403. That is, a calculation for increasing the score is performed for an area where there is no change in the histogram. In step S1405, the in-vehicle image processing apparatus 2000 detects a small area whose score exceeds a predetermined threshold, and stores information (coordinate values) regarding the small area as a recognition result SRD [d] of the foreign object recognition unit. Store in the department. Here, SRD is coordinate information of a dirty small area, and d is an ID when a plurality of coordinates are detected.

[White line determination part]
Next, the contents of processing relating to an example of the white line determination unit 2061 will be described with reference to FIGS. 16, 17, and 18. FIG. 16 is a flowchart showing the processing flow of the white line determination unit 2061 in the present embodiment. First, in step S1501, the first reliability LRLB1 [n] of the white line candidate LC [n] is calculated using the road surface luminance reference. Hereinafter, a method of calculating the first reliability LRLB1 [n] will be described.

  17 and 18 are flowcharts showing the flow of processing for calculating the first reliability LRLB1 [n]. Of each process, the same number is attached | subjected to the location similar to the white line determination part 1061 in the 1st Example shown to FIGS. 7-1 and FIGS. 8-1, and the detailed description is abbreviate | omitted.

  In step S7032, an error Err_LNGP between the luminance histogram HSTLC [n] inside the white line candidate LC [n] and the mixed Gaussian distribution model GP (G) set by the determination method setting unit 1041 is calculated.

  Next, in step S7042, the first reliability LRLB1 [n] of the white line is calculated based on the error Err_LNGP. In this embodiment, the threshold values TH_LNGP1 and TH_LNGP2 of the error Err_LNGP are set to satisfy TH_LNGP1 <TH_LNGP2, and if the error Err_LNGP <TH_LNGP1, the first reliability LRLB1 [n] = 0.0 and the error Err_LNGP ≧ LNGPLG If there is, the first reliability LRLB1 [n] = 1.0, and 0.0 to 1.0 is set so as to perform linear interpolation in the meantime.

  FIG. 18 shows processing in this embodiment when it is determined that the road surface is [complex state]. If it is determined in step S701 that the road surface is in a complicated state, the process moves to step S7062, and the process shown in FIG. 18 is performed.

  First, in step S801, a luminance pattern inside a white line candidate is acquired. In step S8022, the brightness pattern of the white line candidate, that is, the radial degree of the image in the region set to include the white line candidate with respect to the overhead image 1015 is calculated. In this embodiment, an edge angle is calculated from an image in an area set to include white line candidates, and an edge angle DE [x] [y] of each pixel, and an angle DC formed by the position of each pixel and the camera installation position. Pixels whose error [x] [y] is within the threshold TH_DEDC are counted, and a ratio RR of the counted pixels in the area is calculated.

  In step S8032, the first reliability LRLB1 [n] of the white line is calculated based on the radial edge ratio RR. In this embodiment, the thresholds TH_RR1 and TH_RR2 of the radial edge ratio RR are set so that TH_RR1 <TH_RR2, and if the radial edge ratio RR <TH_RR1, the first reliability LRLB1 [n] = 1.0. If the ratio RR> TH_RR2 of the radial edges, the first reliability LRLB1 [n] = 0.0, and 0.0 to 1.0 is set so as to perform linear interpolation in the meantime. With the above processing, the first reliability LRLB1 [n] is calculated in step S1501 of FIG.

  In step S1502, the second reliability LRLB2 [n] of the white line is calculated using the result of the dirt detection unit 2091. The second reliability LRLB2 [n] is calculated from the positional relationship between the detected white line candidate LC [n] and the dirty small region SRD [d]. In the present embodiment, if a dirty small region SRD [d] exists at the position of the white line candidate LC [n], the second reliability LRLB2 [n] = 0 is set, and the dirt closest to the white line candidate LC [n] is set. If the distance D_LCD to the small area SRD [d] is larger than the threshold value TH_LCD, the second reliability LRLB2 [n] = 1.0. If the distance D_LCD is less than the threshold value TH_LCD, 0.0 to 1.0 is set according to the distance. Set to perform linear interpolation.

  Next, in step S1503, the white line reliability LRLB [n] is calculated using the first reliability LRLB1 [n] and the second reliability LRLB2 [n]. In this embodiment, the average value of the first reliability LRLB1 [n] and the second reliability LRLB2 [n] is used. After the calculation, the white line reliability LRLB [n] is output as information accompanying the white line candidate LC [n], and the processing in the white line determination unit 2061 is terminated.

[Parking frame recognition unit]
Next, the contents of processing relating to an example of the parking frame recognition unit 2071 will be described with reference to FIG.

  FIG. 19 is a flowchart showing a processing flow of the parking frame recognition unit 2071 in the present embodiment. Of each process, the same number is attached | subjected to the location similar to the parking frame recognition part 1071 in the 1st Example shown in FIG. 10, and the detailed description is abbreviate | omitted. In the parking frame recognition unit 1071 in the first embodiment, the white line LN [m] is targeted, but in this embodiment, the white line candidate LC [n] is targeted.

  Steps S1901 and S1902 are processes added in the second embodiment. In step S1901, it is determined whether the reliability LRLB [nL] and LRLB [nR] of the two selected lines LC [nL] and LC [nR] are both equal to or greater than the threshold ThRLB. Moves to step S102, and if it is equal to or smaller than the threshold value, moves to step S106.

  In step S1902, not only the coordinate information of the four corners of the frame but also the white line reliability having a smaller value is registered as the parking frame reliability PSRLB [k] in the parking frame PS [k]. The reliability is used by a parking assistance control unit 2081 described later.

  Next, processing of the parking support control unit 2081 will be described. FIG. 20 is a processing flow of the parking assistance control unit 2081. Of each process, the same number is attached | subjected to the location similar to the parking assistance control part 1081 in the 1st Example shown in FIG. 13, and the detailed description is abbreviate | omitted. Steps S201, S202, and S203 are processes added in the second embodiment.

  In step S201, it is determined whether the reliability PSRLB [k] of the acquired parking frame PS [k] is greater than or equal to a threshold value TH_PSRLB. If it is equal to or greater than the threshold value TH_PSRLB, the process moves to step S202, and if it is smaller than the threshold value TH_PSRLB, the process moves to step S203.

  In step S202, strong control is performed on a parking lot with high reliability. In the present embodiment, the steering, the accelerator, and the brake are all controlled to assist the parking in the target parking frame. In step S203, weak control is performed on a parking lot with low reliability. In the present embodiment, any one of steering, accelerator, and brake is controlled to support parking in the target parking frame.

  As described above, the reliability is calculated from the determination result according to the present invention, similarly, the reliability is also calculated from another determination result of the dirt determination, and these are integrated, so that the parking assistance is performed according to the reliability. The degree can be changed.

<Third embodiment>
Next, a third embodiment of the in-vehicle image processing apparatus of the present invention will be described below with reference to the drawings.

  FIG. 21 is a block diagram illustrating a configuration of an in-vehicle image processing device 3000 according to the third embodiment. In the following description, only the parts different from the in-vehicle image processing apparatus 1000 in the first embodiment and the in-vehicle image processing apparatus 2000 in the second embodiment will be described in detail, and the same numbers will be used for similar parts. The detailed description is omitted.

What is characteristic in the present embodiment is that an obstacle is a detection target.
The in-vehicle image processing device 3000 is incorporated in a camera device mounted on an automobile, an integrated controller, or the like, and is used for detecting an object from images captured by the cameras 1001 to 1004. In the form, it is configured to detect an obstacle ahead of the host vehicle.

  The in-vehicle image processing device 3000 is configured by a computer having a CPU, a memory, an I / O, and the like, and a predetermined process is programmed to repeatedly execute the process at a predetermined cycle.

  As shown in FIG. 21, the in-vehicle image processing device 3000 includes an image acquisition unit 1021, a road surface state estimation unit 1031, a determination method setting unit 1041, an obstacle candidate detection unit 3051, an obstacle determination unit 3061, The vehicle control unit 3081 is also included.

  As shown in FIG. 2, the image acquisition unit 1021 acquires images 1011 to 1014 obtained by photographing the surroundings of the vehicle from cameras 1001 to 1004 attached at positions where the surroundings of the vehicle can be imaged, and performs geometric transformation / combination. By doing so, a bird's-eye view image 1015 looking down from the sky virtual viewpoint is generated and stored on the RAM. It is assumed that geometric transformation / synthesis parameters are set in advance by calibration performed at the time of vehicle shipment.

  The road surface state estimation unit 1031 acquires the luminance histogram HST [b] from the own vehicle vicinity region 1032 in the overhead image 1015, and estimates the road surface state. In this embodiment, four types of road surface conditions are estimated: [uniform state], [bright / dark separation state], [shade-sunshine state], and [complex state].

  The determination method setting unit 1041 sets a determination method for determining a white line according to the road surface state estimated by the road surface state estimation unit 1031. In the present embodiment, in the case of [uniform state], [brightness / darkness separation state], and [shade-sunshine state], determination is performed using the luminance range obtained from the luminance histogram or the Gaussian model of the luminance histogram, and [complex state] ], Determination is made using an image pattern.

  The obstacle candidate detection unit 3051 detects an obstacle candidate OBJC [i] from the overhead image 1015. The obstacle determination unit 3061 determines an obstacle OBJ [j] for the obstacle candidate OBJC [i] based on the determination criterion set by the determination method setting unit 1041. Further, the vehicle control unit 3081 is mounted in a control device mounted on the same vehicle as the in-vehicle image processing device 3000, and is based on the position information of the obstacle OBJ [j] received from the in-vehicle image processing device 3000. In order to prevent collisions, warnings and brake control are implemented.

[Obstacle candidate detection unit]
The contents of processing relating to an example of the obstacle candidate detection unit 3051 will be described.
The obstacle candidate detection unit 3051 in the present embodiment has two patterns when using a sensor other than an image and when using an overhead image 1015. When using a sensor other than an image, for example, a sonar sensor or the like is used to detect an obstacle and acquire position information of the obstacle. Next, the position on the bird's-eye view image 1015 is specified using the camera geometric information. In addition, obstacles such as pedestrians can be directly detected from the image using, for example, a known pattern matching technique. Also in this case, the position on the bird's-eye view image 1015 is specified using the camera geometric information.

[White line judgment part]
Next, contents of processing in the obstacle determination unit 3061 will be described with reference to FIGS.

FIG. 22 is a flowchart showing a process flow of the obstacle determination unit 3061.
First, in step S2201, it is confirmed whether or not the road surface determined by the road surface state estimation unit 1031 is [complex state]. If it is not a complicated state, the process proceeds to step S2202, and if it is a complicated state, the process proceeds to step S2206. Hereinafter, the processing from step S2202 to step S2205 is repeated by the number of obstacle candidates LC [i].

  First, in step S2202, a luminance histogram HSTABJC [i] inside the obstacle candidate OBJC [i] is created. Next, in step S703, an error Err_OBJGP between the luminance histogram HSTABJC [i] inside the obstacle candidate OBJC [i] and the mixed Gaussian distribution model GP (G) set by the determination method setting unit 1041 is calculated. And compare with the threshold TH_BOJGP. If the error Err_OBJGP is greater than the threshold value TH_BOJGP, it is determined that it is not included in the road surface brightness reference, and the process proceeds to step S2204.

  In step S2204, the obstacle candidate is determined to be an obstacle, and the information of the obstacle candidate OBJC [i] is copied to the obstacle OBJ [j]. In step S2205, it is determined that the obstacle candidate is not an obstacle, and the information on the obstacle candidate OBJC [i] is deleted without being copied to the obstacle OBJ [j]. In step S2206, the process when it is determined that the state is a complex state described below is entered.

  FIG. 23 shows a process when it is determined that the road surface state is [complex state]. The processing from step S2301 to step S2304 is repeated by the number of obstacle candidates OBJC [i].

  First, in step S2301, a region including only an obstacle candidate is set for the luminance pattern around the obstacle candidate, that is, the overhead image 1015. Next, in step S2302, it is determined whether the image in the region set to include only the obstacle candidate with respect to the overhead image 1015 is radial. As described with reference to FIG. 2, the bird's-eye view image 1015 is a virtual viewpoint image from the sky. Therefore, when the pedestrian 20 exists, the pedestrian 20 shown in FIG. 24A in the image before the overhead conversion is reflected radially as shown in FIG. Since this pattern extends radially from the camera installation position on the bird's-eye view image 1015, it is possible to determine whether or not the image within the region set to include only the obstacle candidate is radial from the camera installation position. It is possible to distinguish whether the object candidate is a three-dimensional object.

  In the present embodiment, as a method of determining whether the pixel is radial, an edge angle is calculated from an image in an area set to include only obstacle candidates, and the edge angle DE [x] [y] of each pixel is calculated. Pixels in which the error of the angle DC [x] [y] between the position of each pixel and the camera installation position is within the threshold TH_DEDC are counted, and whether or not the ratio RROBJ that the counted pixels occupy in the area of the area is greater than or equal to the threshold TH_RROBJ Judgment accordingly. The threshold values TH_DEDC and TH_RROBJ are set in advance.

  If RROJ is greater than or equal to the threshold TH_RROBJ in step S2302, it is determined that it is not radial, and if it is less than the threshold, it is determined that it is radial, and the process moves to step S2303.

  In step S2303, the obstacle candidate is determined to be an obstacle, and the information of the obstacle candidate OBJC [i] is copied to the obstacle OBJ [j]. In step S2304, it is determined that the obstacle candidate is not an obstacle, and the information of the obstacle candidate OBJC [i] is not copied to the obstacle OBJ [j] but is deleted.

  As described above, whether or not the obstacle candidate OBJC [i] is included in the obstacle OBJ [j] in the obstacle determination unit 3061 based on the determination criterion according to the luminance of the obstacle candidate OBJC [i] and the road surface condition. Explained the process. In this embodiment, when the road surface state is [uniform state] or [multi-color road surface], the luminance histogram HSOBJC [b] inside the obstacle candidate OBJC [i] and the mixed Gaussian distribution approximation model GP (G) It is determined whether the obstacle candidate is an obstacle based on the magnitude of the error. Further, when the road surface is [complex state], the determination is made based on the degree to which the luminance pattern of the region including only the obstacle candidate OBJ [i] has a radial feature. By such processing, it becomes possible to make a determination according to the road surface condition, and to prevent erroneous recognition of an obstacle.

[Vehicle control unit]
Next, the contents of processing in the vehicle control unit 3081 will be described with reference to FIG.
First, in step S2501, relative distances PX [j] and PY [j] from the vehicle are acquired from the obstacle information OBJ [j] from the in-vehicle image processing device 3000. The obstacle information OBJ [j] may be acquired by directly inputting a signal from the in-vehicle image processing device 3000 to the control device, or may be acquired by performing communication using a LAN.

  Hereinafter, steps S2502 to S2505 are repeated for the number of obstacles. In step S2502, the relative speed VX with the obstacle calculated by the difference approximation between the obstacle positions PX [j] and PY [j] and the obstacle positions PX_z1 [j] and PY_z1 [j] of the previous cycle. [J] and VY [j] are calculated.

  Next, in step S2503, it is determined whether the relative speeds VX [j] and VY [j] are approaching the host vehicle. This determination is made, for example, when a coordinate system with the vehicle center as the origin is taken into consideration as a coordinate system, and the velocity vector between the obstacle positions PX [j], PY [j] and the straight lines connecting the four corners of the vehicle. This can be determined by looking at the orientation. If the vehicle is heading toward the host vehicle, the process proceeds to step S2504. If the vehicle is not heading, the process proceeds to step S2505.

In step S2504, the collision time TTC [j] is calculated. TTC [j] is a parameter indicating how many seconds after the own vehicle collides with the obstacle. In this case, since only an object heading toward the host vehicle is handled in step S2503, it can be obtained from the following equation.
PL [j] = SQRT (PX [j] × PX [j] + PY [j] × PY [j])
VL [j] = SQRT (VX [j] × VX [j] + VY [j] × VY [j])
TTC [j] = PL [j] / VL [j]

Here, SQRT () is a function for calculating the square root.
In step S2505, since there is no danger of a collision, a sufficiently large value is substituted for the collision time TTC [j]. In this embodiment, TTC [j] = 10000.

  Next, in step S2506, the minimum collision time TTCMIN is extracted from TTC [j]. Next, in step S2507, the minimum collision time TTCMIN is compared with the threshold value TH_TTCBRK. If the minimum collision time TTCMIN is less than or equal to the threshold value TH_TTCBRK, the process moves to step S2508 to execute vehicle brake control. On the other hand, if the minimum collision time TTCMIN is greater than the threshold value TH_TTCBRK, the process moves to step S2509.

  In step S2509, the minimum collision time TTCMIN is compared with the threshold value TH_TTCALM. When the minimum collision time TTCMIN is less than or equal to the threshold value TH_TTCALM, the process moves to step S2510, the vehicle alarm device is operated, and the driver is alerted. On the other hand, if the minimum collision time TTCMIN is greater than the threshold value TH_TTCALM, the processing of the vehicle control unit 3081 is terminated. As described above, the alarm and the brake control can be activated according to the obstacle information detected by the in-vehicle image processing device 3000.

  Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various designs can be made without departing from the spirit of the present invention described in the claims. It can be changed. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to one having all the configurations described. Further, a part of the configuration of an embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of an embodiment. Furthermore, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

1000 In-vehicle image processing devices 1001 to 1004 Camera 1021 Image acquisition unit 1031 Road surface state estimation unit 1041 Determination method setting unit 1051 White line candidate detection unit 1061 White line determination unit 1071 Parking frame recognition unit 1081 Parking support control unit 2000 In-vehicle image processing device 2091 Dirt detection unit 2061 White line determination unit 2071 Parking frame recognition unit 2081 Parking support control unit 3000 In-vehicle image processing device 3051 Obstacle candidate detection unit 3061 Obstacle determination unit 3081 Vehicle control unit

Claims (7)

An in-vehicle image processing device that detects an object on a road surface,
An image acquisition unit for acquiring an image of the surroundings of the vehicle;
A road surface state estimation unit for estimating which road surface state around the vehicle belongs to one of a plurality of preset road surface states based on the luminance distribution in the vehicle vicinity region of the image;
A determination method setting unit that sets a determination method of an object according to the estimated road surface state;
An object candidate detection unit for detecting an object candidate estimated as the object from the image;
An object determination unit that determines whether or not the object candidate is the object using the determination method;
I have a,
The road surface state estimation unit estimates whether the road surface state is a complex state of the plurality of road surface states based on whether the luminance distribution can be approximated by a Gaussian distribution of 1 or 2.
The determination method setting unit sets a determination method for determining based on whether or not the brightness pattern of the target object is radial when the road surface state estimation unit estimates that the road surface state is the complex state. An in-vehicle image processing apparatus. The road surface state estimation unit determines whether the luminance distribution of the image is a uniform state that can be approximated to a single Gaussian distribution having a peak of 1,
The determination method setting unit sets a determination method for determining the object based on a single luminance range as a determination criterion from the luminance distribution when it is determined that the road surface state is a uniform state. The in-vehicle image processing apparatus according to claim 1 . The road surface state estimation unit determines whether or not the luminance distribution of the image can be approximated to a mixed Gaussian distribution having two peaks, and determines that the road surface state cannot be approximated to the mixed Gaussian distribution. The in-vehicle image processing apparatus according to claim 2 , wherein the in-vehicle image processing apparatus is estimated to be in a state. The road surface state estimation unit determines whether the road surface state is a reflection state or a light / dark separation state based on a road surface pattern when the luminance distribution of the image can be approximated to a mixed Gaussian distribution having two peaks. ,
The determination method setting unit, wherein said road surface condition and sets a determination method based on whether luminance pattern of the object candidate or radially when it is determined that the a reflective state Item 4. The on-vehicle image processing device according to Item 3 . The road surface state estimation unit performs a labeling process on a binary image obtained by extracting pixels belonging to a Gaussian distribution having a low average luminance in the mixed Gaussian distribution, and looks at the shape of each label to thereby determine the road surface state. The vehicle-mounted image processing apparatus according to claim 4 , wherein the image processing apparatus determines whether the state is a reflection state or a light / dark separation state. When the road surface state estimation unit determines that the road surface state is the light / dark separation state, the road surface state determination unit determines whether or not the road surface state is a shaded and sunny state in which a shadow of the vehicle is reflected in the image,
The determination method setting unit sets a determination method for determining the object based on a predetermined sun road surface luminance reference and a shade road surface luminance reference as a determination reference when the shade method is determined to be in the shaded sun state.
When it is determined that the object determination unit is in the shaded and sunny state, the object method includes an inner luminance distribution of the object candidate included in the sun road surface luminance reference, and the object candidate. Is determined to be not the target object, and the inner luminance distribution of the target candidate is not included in the sunny road surface brightness reference. 6. The vehicle-mounted image according to claim 5 , wherein when the outer luminance distribution of the object candidate is not included in the shade road surface luminance reference, the object candidate is determined to be the object. Processing equipment. A lens dirt detector for detecting the position of the camera lens dirt,
When the road surface state estimation unit estimates that the road surface state is a complex state, based on the ratio of radial edges in the inner luminance distribution of the object candidate, when it is not estimated to be a complex state A first reliability calculation unit that calculates a first reliability based on an error between the inner luminance distribution of the object candidate and a mixed Gaussian distribution model ;
A second reliability calculation unit that calculates a second reliability based on the lens contamination position detected by the lens contamination detection unit ;
The on-vehicle object according to claim 1, wherein the object determination unit determines whether or not the object candidate is the object based on the first reliability and the second reliability. Image processing device.

Images