JPH11339012A - Image processor for vehicle - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make sure to follow a white line even when an image processor is used for an automatic parking device, a parking guiding device, etc., by setting the image processing area based on measured run amount of a vehicle. SOLUTION: Images of the ambient environment of a vehicle which are picked up by a side camera C1 and a rear camera C2 provided on the vehicle are stored in an image memory 3 with a window set at a part of an image pickup screen by a window setting part 9. A white line processing part 4 performs image processing that extracts a white line which exists in the ambient environment of the vehicle and whose position does not change relatively to the vehicle with running of the vehicle. In such a case, a vehicle run amount calculating part 6 calculates the run amount of the vehicle according to a measurement result of a wheel speed sensor 2 and sets the window to the image pickup screen after the vehicle travels based on the vehicle travel amount. It is possible to perform automatic parking and driving by a driver while always recognizing the accurate position its own vehicle by repeating such processing.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for a vehicle for extracting a white line of a road or a parking lot.

[0002]

2. Description of the Related Art As a conventional image processing apparatus for a vehicle, for example, there is one as disclosed in Japanese Patent Application Laid-Open No. Hei 5-303625. This vehicular image processing device is a white line extracting device that extracts a white line on a road, performs predetermined image processing on a plurality of windows that are continuously set along a white line existing in an image, and generates a white line. Image processing means for extraction, window coordinate calculation means for calculating the center position in the width direction of the white line extracted by the image processing means, and window coordinate storage means for storing and holding the calculated center position. Then, based on the center position information of the white line stored in the window coordinate storage unit, the window is set so that the center of the window in the next image is aligned with the center position of the white line, thereby performing the processing associated with the window setting. It is trying to reduce the amount and shorten the processing time.

[0003]

However, in such a conventional image processing apparatus for a vehicle, a change in the position of a white line in an imaging screen is relatively small, such as when traveling straight on a road. When it is small, it can follow the white line,
When an image processing device for a vehicle is used for monitoring parking position information in an automatic parking device or a parking guidance device, the position and orientation of the own vehicle greatly changes with respect to the white line of the garage frame, so that the white line on the imaging screen is changed. Since the position and orientation greatly fluctuate, there is a problem that it is difficult to follow the white line.

[0004] Further, if the white line once deviates from the field of view of the camera due to large fluctuations in the position and attitude of the vehicle, there is a problem that even if the white line enters the field of view of the camera again, it cannot be followed at all. there were.

An object of the present invention is to use the present invention in, for example, an automatic parking system or a parking guidance system, even when the position and posture of the vehicle fluctuate greatly with respect to a road or a white line in a parking lot. An object of the present invention is to provide a vehicular image processing device that can reliably follow a white line even in such a case.

[0006]

DETAILED DESCRIPTION OF THE INVENTION The present invention will be described below with reference to the embodiment shown in FIG. In order to achieve the above object, the invention according to claim 1 includes an imaging unit (C1, C2) provided in a vehicle for imaging an environment around the vehicle, and an imaging unit (C1, C2)
An image processing area setting means (9) for setting an image processing area for performing image processing on a part of the imaging screen imaged in 2); and an image processing area setting means (9) existing in a surrounding environment of the vehicle and moving relative to the vehicle as the vehicle moves. Image processing means for performing image processing in the image processing area set by the image processing area setting means (9) in order to extract an object whose position changes and the position does not change relatively to the moving surface of the vehicle 3 and 4), and further includes a vehicle movement amount measuring means (2, 6) for measuring a movement amount on a moving surface of the vehicle, and an image processing area setting means (9). Is to set the image processing area after the movement of the vehicle based on the movement amount of the vehicle measured by the vehicle movement amount measuring means (2, 6). According to a second aspect of the present invention, in the vehicle image processing apparatus according to the first aspect, the vehicle after the movement is based on the positional relationship of the object extracted by performing image processing on the image processing area set after the movement of the vehicle. Is recognized. According to a third aspect of the present invention, in the image processing apparatus for a vehicle according to the first aspect, the imaging unit includes a plurality of imaging devices (C1, C2), and the image processing region setting unit (9) is a vehicle movement amount measurement unit. A plurality of imaging devices (C1, C2) for setting an image processing area after moving the vehicle based on the movement amount of the vehicle measured by (2, 6)
One of them is selected. Claim 4
The invention according to claim 1, wherein in the image processing apparatus for a vehicle according to claim 1, when the vehicle is at a predetermined position on the moving surface, a predetermined position on the moving surface is set as a reference coordinate, and an imaging screen of the imaging device is set as an imaging screen coordinate. Coordinate conversion means (5, 7) for mutually transforming the reference coordinates and the imaging screen coordinates in consideration of the movement amount of the vehicle from a predetermined position measured by the vehicle movement amount measurement means (2, 6); In addition, the coordinate conversion means (5, 7) converts the coordinates of the object extracted on the imaging screen on the imaging screen coordinates into reference coordinates. According to a fifth aspect of the present invention, in the image processing apparatus for a vehicle according to the fourth aspect, the image processing area setting means (9) sets an area including the target object extracted on the reference coordinates,
Thereafter, when the vehicle moves, the image processing area on the imaging screen is set by performing coordinate conversion of the area into the imaging screen coordinates by the coordinate conversion means (5, 7). According to a sixth aspect of the present invention, an image pickup means (C1, C2) provided in the vehicle for picking up an image of the environment around the vehicle, and an image processing device for performing image processing on a part of the image picked up by the image pickup means (C1, C2). An image processing area setting means (9) for setting an image processing area; a position existing in a surrounding environment of the vehicle, the position of which changes relative to the vehicle as the vehicle moves, and the position of which changes relative to the moving surface of the vehicle Image processing means (3, 3) for performing image processing in the image processing area set by the image processing area setting means (9) in order to extract an object not to be processed;
4) a vehicle trajectory setting means for setting a trajectory to which the vehicle should travel, and an image processing area based on vehicle position information at one or more points on the trajectory. And a trajectory planning means (8) for planning in advance whether to set the trajectory. According to a seventh aspect of the present invention, in the image processing apparatus for a vehicle according to the sixth aspect, the image pickup means includes a plurality of image pickup apparatuses (C1, C2), and the plurality of image pickup apparatuses are included in the plan by the track planning means (8). This includes a plan for selecting which imaging device (C1, C2) to select.

[0007] In the means for solving the above-mentioned problems, correspondence is made with the drawings of the embodiment for easy understanding, but the present invention is not limited to the embodiment.

[0008]

Since the present invention is configured as described above, it has the following effects. According to the first aspect of the present invention, the image processing area (window area) is set based on the measured movement amount of the vehicle. Even if the position and the attitude of the vehicle change greatly, the white line can be reliably followed. Further, even when the white line once deviates from the field of view of the camera due to a large change in the position and attitude of the own vehicle, it is possible to reliably follow the white line again when it enters the field of view of the camera. According to the invention of claim 2, it is possible to accurately recognize the position of the vehicle after moving. According to the invention of claim 3, when a plurality of different cameras are used, even if the white line deviates from the field of view of one camera, it is possible to easily determine which of the other cameras is in the field of view, The white line can be reliably followed.
According to the fourth aspect of the present invention, the white line extracted on the reference coordinate can be easily recognized by using the coordinate conversion technique. According to the fifth aspect of the present invention, the image processing area (window area) set on the reference coordinates can be easily set on the imaging screen by using the coordinate conversion technique. According to the invention of claim 6, when the trajectory of the vehicle is known in advance, an image processing area (window area) can be set in advance, so that a strategy for efficient and efficient image processing control can be established. . According to the invention of claim 7, when the trajectory of the vehicle is known in advance and a plurality of cameras are used, it is possible to calculate in advance which of the cameras the white line falls within the visual field of the route in the middle of the trajectory, An efficient and lean image processing control strategy is established.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A vehicular image processing apparatus according to this embodiment processes images taken by two cameras attached to a vehicle and extracts white lines drawn on a road or a parking lot. Used for automatic parking equipment.

FIG. 1 is a diagram showing a configuration of an embodiment of a vehicle image processing apparatus according to the present invention. Reference numeral 1 denotes a control device that controls the image processing apparatus for a vehicle, and includes a microprocessor and peripheral circuits, and executes a predetermined program. C1 is a camera provided on the side of the vehicle, and C2 is a camera provided behind the vehicle. The cameras C1 and C2 are image pickup devices for picking up an image of the environment around the vehicle, and are constituted by, for example, a CCD. Reference numeral 2 denotes wheel speed sensors provided on both left and right rear wheels for measuring a moving amount of the vehicle.

The control device 1 has an image memory 3 for storing image data captured by the cameras C1 and C2.
Further, the control device 1 executes a predetermined program corresponding to each function, and is composed of functional blocks as shown in FIG. Reference numeral 4 denotes a white line processing unit for performing image processing of the image data in the image memory 3 and extracting white lines. Reference numeral 5 denotes a window coordinate calculator for calculating a window area on the screen in the cameras 1 and 2. Reference numeral 6 denotes a vehicle movement amount calculation unit that calculates the vehicle movement amount from the measurement result of the wheel speed sensor 2. The vehicle moving amount calculating unit 6 calculates the moving amount of the vehicle from the angular velocity of the rear wheel detected by the wheel speed sensor 2 and the wheel diameter, tread, and the like input in advance. 7
Is a vehicle coordinate calculator for calculating coordinates in the vehicle coordinate system. Reference numeral 8 denotes a trajectory planning unit for generating a trajectory of the vehicle. Reference numeral 9 denotes a window setting unit for setting a window in the imaging screen.

Next, the definition of the coordinate system and the relational expressions used in the present embodiment will be described. FIG. 2 is a plan view when the vehicle A is viewed from above. A camera C1 is provided on the side, and a camera C2 is provided on the rear. The vehicle coordinate system taking the axle center V to the origin after the vehicle and sigma _V. Lateral camera C1 is, l to Y _V axis direction of the vehicle coordinate system sigma _V
1 and X _V-axis direction is provided at a distance of w _1, the rear camera C2, the vehicle coordinate system sigma _V of Y _V axis direction l ₂ and X _V
Axially disposed at a distance of w _2. FIG. 3 is a view of the vehicle A as viewed from the rear. The camera coordinate system taking the lens center of the camera C1 to the origin as shown in FIG. 3 and sigma _C1. The camera C1 is provided at a position of a predetermined height h _1,
Are inclined in a direction to image a road surface at an angle theta _1. FIG.
Is a view of the rear part of the vehicle A viewed from the right side. The camera coordinate system taking the lens center of the camera C2 to the origin as shown in FIG. 4 and sigma _C2. The camera C2 is provided at a position of a predetermined height h _2, are inclined in a direction to image a road surface at an angle theta _2.

FIG. 5 shows a vehicle coordinate system { _V and a camera coordinate system}.
FIG. 4 is a diagram illustrating a relationship between positions of an object P viewed from _C1 . Position of the object P viewed from the vehicle coordinate system ^{_{Σ V V r = [V X}} , V Y, V
Z] ^T and the position of the object P viewed from the side camera coordinate system _C1
Relationship ^{C1 r = [C1 X, C1} Y, C1 Z] T is given by the following equation (1).

(Equation 1) Here, ^V _PC1 is the following equation (2):

(Equation 2) ^VR _C1 is given by the following equation (3).

(Equation 3) Replacing the first formula (1) ^V r as follows first (1A) equation,

(Equation 4) Equation (1) is expressed in the form of the following equation (4).

(Equation 5) However, ^V _TC1 is represented by the following equation (5).

(Equation 6) This equation (5) is the vehicle coordinate system Σ _V and the side camera coordinate system Σ
It is an expression showing a geometric relationship between _C1 . Although the bold characters r, R, P, and T in the above equations represent vectors, the same vectors are also used for the corresponding non-bold characters r, R, P, and T in the description. The same applies to the following description.

[0014] in the same manner as described above, the position ^V r = the object P viewed from the vehicle coordinate system _{^{Σ V [V X, V Y}} , V Z] T and the rear camera coordinate system sigma _C2 of the object P viewed Position ^C2 r = [ ^C2 X, ^C2
Y, ^C2 Z] ^T is given by the following equation (6).

(Equation 7) Here, ^V _PC2 is expressed by the following equation (7):

(Equation 8) ^V R _C2 is given by the following equation (3).

(Equation 9) Replacing equation (6) of ^V r like first (1A) equation described above, equation (6) can be expressed in the form of the following first equation (9).

(Equation 10) Here, ^V _TC2 is expressed by the following equation (10).

[Equation 11] The first (10) is an expression showing the geometric relationship between the vehicle coordinate system sigma _V and the rear camera coordinate system sigma _C2.

[0015] Next, the relationship between the camera coordinate system sigma _C and photographing (imaging) the screen coordinate system. 6 illustrates the relationship between the camera coordinate system sigma _C shooting screen coordinate system. f is the focal length of the camera lens. From this figure, the side camera C
The position of the object ^C1p = [ ^C1x , ^C1y ] ^T on the shooting screen 1
Each element of the next first using elements of the lateral position of the object P viewed from the camera coordinate system ^{_{Σ C1 C1 r = [C1 X}} , C1 Y, C1 Z] T (11) equation, the It is given by equation (12). Where f ₁ is the focal length of the lens of the camera C1.

(Equation 12)

(Equation 13) Similarly, each element of the position ^{C2 p = [C2 x, C2} y] T of the object on the imaging screen of the rear camera C2, the position of the object P viewed from the rear camera coordinate system sigma _C2 ^C2 r = [ ^C2 X, ^C2 Y, ^C2
Z] Using each element of ^T , the following equation (13),
It is given by equation 4). Here f ₂ is the focal length of the lens of the camera C2.

[Equation 14]

(Equation 15)

Next, the relationship between the vehicle coordinate system ΣV _(t0) before the movement and the vehicle coordinate system ΣV _(t1) after the movement is considered. FIG. 7 is a diagram illustrating the relationship. Vehicle A moves from time t = t ₀ to t = t ₁ as shown in FIG.
_{^{V (t0) P V (t1}} ) = [Δx (t 0; t 1), Δy (t 0;
t _1), 0] ^T as the vehicle coordinate system sigma origin Δθ counterclockwise around ^V Z-axis by the movement of the _V (t _0; t ₁₎ vehicle coordinate system when pivoted Σ _{V (t0)} and sigma _V The relationship of _(t1) is given by Expression (15). Here, ^{V (t0)} P _{V (t1)} is the following equation (16): ^{V ( t0)}
R _{V (t1)} is expressed by equation (17).

(Equation 16)

[Equation 17]

(Equation 18) The following equation (18) is established in the same manner as described above. Here, ^{V (t0) TV} _(t1) is expressed by the following equation (19).

[Equation 19]

(Equation 20)

Next, measurement of the amount of movement of the vehicle will be described. The amount of movement of the vehicle is determined as follows by detecting the angular velocities of the left and right wheels by the wheel speed sensors 2 mounted on the left and right rear wheels. Set the angular velocity of the left and right wheels to u
_l (t), and u _r (t), the left and right wheel diameter _{R l,} R _r, translational speed s of the vehicle when the tread and T (t) and the angular velocity omega (t) following the are (20) It is given by the equation and the equation (21). _{s (t) = (R r} · u r (t) + R l · u l (t)) / 2 ···· (20) ω (t) = (R r · u r (t) -R l · u _l (t)) / T (21) The position at time t = t ₀ is defined as P (t ₀ ) = [0,0,0,
0] T, the position P (t _{0 at} time t = t ₀ + τ
+ Τ) is given by equation (22).

(Equation 21) The expression (22) ^{V (t0)} by repeated from t = t ₀ to t = t ₁ to P _{V (t1)} and Δθ (t _0; t ₁₎ is obtained. That is, the movement amount and the vehicle body rotational angle of the axle center V after the vehicle A is found at time t ₁ as seen from the vehicle coordinate system at time t ₀ Σ _{V (t0).}

Next, on the screen of the side camera C1, [ ^C1x ,
^C1 y] which position in the vehicle coordinate system sigma _V of the target point P on the road surface that is on ^{T V} r = ^[V X, determine whether in ^V Y, 0] ^T. From equations (11) and (12), equations (23) and (24) hold in the camera coordinate system.

(Equation 22)

(Equation 23) From the expressions (4), (5), (23), and (24), the following expression (25) is established.

(Equation 24) Comparing both sides with respect to equation (25), the following equation (26)
Expressions (27) and (28) are obtained.

(Equation 25)

(Equation 26)

[Equation 27] From equation (28), ^C1Z is obtained.
^{V r = [V X, V} Y, 0] is substituted into the equation (27) is ^T determined.

Next, at time t = t ₀ , the vehicle coordinate system Σ _V
Viewed from position ^{V r = [V X, V} Y, 0] V (t0) to a target the time t = t ₁ to _{^{T P V (t1) = [}} Δx (t 0; t 1), Δ
_{y (t 0; t 1)} , 0] T as the vehicle coordinate system Σ origin Δθ counterclockwise around ^V Z-axis by the movement of the _V (t _0; t ₁₎ subject to lateral camera C1 when turning consider the position on the screen of the ^{[C1 (t 1) x,} C1 (t1) y] T. From the expressions (4) and (18), the following expression (29) is established,

[Equation 28] ^{C1 (t1)} r = [ ^{C1 (t1)} X, ^{C1 (t1)} Y, ^{C1 (t1)} Z] ^T is obtained, and [ ^{C1 (t1)} x, ^C1 is obtained from the equations (11) and (12). ^(t1) y] ^T is obtained. Note that the second (2)
_{^{9) [V T C1] -1}} [V (t0) T V (t1)] in Formula ^-1 can be expressed by the equation (30) below.

(Equation 29)

Similarly, for the rear camera C2, the following Expression (31) is established from Expressions (9) and (18).

[Equation 30] ^{C2 (t1)} r = [ ^{C2 (t1)} X, ^{C2 (t1)} Y, ^{C2 (t1)} Z] ^T is obtained, and [ ^{C2 (t1)} x, ^{C2 (t1) (t1)} y] ^T is obtained. In addition, (3
_{^{1) [V T C2] -1}} [V (t0) T V (t1)] in Formula ^-1 can be expressed by the equation (32) below.

(Equation 31)

Based on the above preparation, an algorithm for setting a window based on the amount of movement of the vehicle, which is a characteristic part of the present embodiment, will be described with reference to flowcharts shown in FIGS. FIG. 8 is a flowchart for making a global plan as to which of a plurality of cameras performs image processing at each point of the track when the track of the vehicle is known in advance at the time of starting automatic parking. FIG. 9 is an algorithm for calculating the window setting area based on the measured movement amount of the vehicle. 8 and 9
Is executed by the control device 1 in FIG.

First, an algorithm for planning which camera performs image processing at each point on the trajectory according to FIG. 8 will be described. Assuming automatic parking, the driver first stops the vehicle in front of the garage to which the driver wants to enter, and then presses the automatic parking start button. This routine starts when this start button is pressed.

In step S10, the image taken by the side camera C1 is taken into the image memory 3. It is assumed that a white line 11 of the parking lot is shown in the side camera C1 as shown in FIG. In step S20, edge extraction is performed on the entire captured image using a sobel filter or the like. In step S30, a set of trapezoidal window regions having a pair of opposite sides parallel to each other is initially set as shown in FIG. 10, and in step S40, the sum of density values of pixels on all straight lines to be detected in the window is calculated. And control device 1
A table is created on a memory (not shown) in the table. In step S50, the maximum value in the table is detected. Step S
60, select the straight line corresponding to the detected maximum value as a straight line most probable as the edge straight, the straight line on two points on the window border of ^{_{[C1 x si, C1 y 1}} ] and ^[C1 x
^{_{ei, C1 y 2] (i}} = 1~2) carries out the search of the predetermined concentration value or more pixels in a direction connecting the to the point at which the pixel is interrupted and end points of the white line of the garage frame (Figure 11). Step S70
In, converts the white line endpoints determined in step S60 to the vehicle coordinate system sigma _V. Figure 12 is a diagram illustrating a white line 11 and its end point in the vehicle coordinate system sigma _V. Step S60
The coordinates of the upper end on the camera screen of the end point obtained in
^C1 x _ui , ^C1 y _ui ], and the coordinates of the lower end are [ ^C1 x _li , ^C1 y _li ] (i =
When 1-2), the position of the vehicle coordinate system sigma _V end point of the white line shown in FIG. ^{_{12 V P ui [V X ui}} , V Y ui, 0] T and ^V P _li [
^V X _li , ^V Y _li , 0] ^T is the above-mentioned equation (26) to (2)
8) It is given by the equation.

[0024] In this way positional relationship between the vehicle coordinate system sigma _V and the white line is recognized, in step S80, the target trajectory of the vehicle A for automatic parking is set are computed. A known method is used for setting the target trajectory for the automatic parking, and the description thereof is omitted here.

The next position ^V of the vehicle coordinate system Σ _V of the white line of the end point
P _ui [ ^V X _ui , ^V Y _ui , 0] ^T and ^V P _li [ ^V X _li ,
^V Y _li , 0] ^T is the side camera C1 at each point on the trajectory.
Position [ ^{C1 (t1)} x _ui ,
^{C1 (t1)} _yui ] ^T , [ ^{C1 (t1)} _xli , ^{C1 (t1)} _yui ] ^T and a position [ ^{C2 (t1)} _xui , existing on the screen of the rear camera C2.
Consider ^{C2 (t1)} _yui ] ^T , [ ^{C2 (t1)} _xli , ^{C2 (t1)} _yui ] ^T. Each point on the track is determined in advance by a predetermined reference according to the shape of the set target track or the like, but may be arbitrarily set by the driver.

In step S90, when the vehicle A is located at a certain point on the track, the vehicle (A)
From equation (31) and equation (31), ^{C1 (t1)} _rui = [ ^{C1 (t1)} _Xui ,
_{^{C1 (t 1) Y ui,}} C1 (t1) Z ui] T and ^{C1 (t1)} r _li = [
^{C1 (t1)} _Xli , ^{C1 (t1)} _Yli , ^{C1 (t1)} _Zli ] ^T are obtained, and [ ^{C1 (t1)} _xui , ^{C1 (t1)} are obtained from the equations (11) and (12 ^). _yui ] ^T , [ ^{C1 (t1)} _xli , ^{C1 (t1)}
_yui ] ^T is required. Similarly, from Expressions (30) and (32), ^{C2 (t1)} _rui = [ ^{C2 (t1)} _Xui , ^{C2 (t1)}
Y _ui , ^{C2 (t1)} Z _ui ] ^T and ^{C2 (t1)} r _li =
_{^{[C2 (t1) X li,}} C2 (t1) Y li, C2 (t1) Z li] T is calculated et al, the (11) and a second (12) than _{^{[C2 (t1) x ui,}} C2 (t1 ⁾ y _ui ] ^T , [ ^{C2 (t1)} _xli , ^{C2 (t1)}
_yui ] ^T is required.

In step S100, [ ^{C1 (t1)} _xui , ^{C1 (t1)} _yui ] ^T obtained in step S90, [ ^{C1 (t1)} _xli ,
^{C1 (t1)} y _ui] ^T imaging screen in the range of the lateral cameras C1 | determine present in _{^{≦ N 1 | C1 x | ≦}} M 1 and | ^C1 y. Similarly, [ ^{C2 (t1)} _xui , ^{C2 (t1)} _yui ] ^T , [
_{^{C2 (t1) x li, C2}} (t1) y ui] T range on the imaging screen of the rear camera ^C2 | C2 x | ≦ M ₂ and | ^C2 y | even if present in ≦ N ₂ examined. In step S110, if none of the four end points is present in the imaging screen of the side camera C1, the white line of the garage frame does not appear on the side camera C1. It is assumed that no image processing is performed on the image of (1), and information to that effect is stored in the data area of the target trajectory. Similarly, if none of the four end points is present in the image screen of the rear camera C2, the white line of the garage frame is not shown on the rear camera C2. No processing,
The information to that effect is stored in the data area of the target trajectory.

In step S120, it is determined whether or not all the above determinations have been completed at each set point on the trajectory. If the determination has been completed, the process is terminated. If not, the process returns to step S90 and repeats the process. .

The determination as to whether or not to perform image processing as described above is made for each appropriately sampled point on the trajectory, and a diagram for performing or not performing image processing for each camera is created and a plan is made. In this way, by making a plan for global image processing of the traveling trajectory of the vehicle, even when a plurality of cameras are used, effective resource utilization can be achieved by efficiently allocating one control device 1.

Next, an algorithm for calculating the window setting area based on the measured vehicle movement will be described with reference to FIG. Position in the vehicle coordinate system sigma _V of the white line of the end point ^{_{V P ui [V X ui,}} V Y ui, 0] T and ^V P
^{_{li [V X li, V Y}} li, 0] because ^T is the same as the contents required in up to step S10~S70 of the algorithm of FIG. 8, in this flow chart, a method of setting the window on the basis of the end points It will be explained first.

FIG. 13 shows the state of the set window.
You. End of white line^VP_ui[^VX_ui,^VY_{u i}, 0]^Tand^VP
_li[^VX_li,^VY_li, 0]^T Unit vector connecting^Ve_iIs
It is given by equation (33).

(Equation 32) ^V e _i perpendicular unit vectors ^V f _i is given by the equation (34) below.

[Equation 33] Therefore, as shown in FIG. 13, the end points ^V _Pqi , ^V _Pri ,
^V P _si, position vector ^V r _qi of _{^{V P ti, V r ri,}} V r si, V r
_ti is given by the following equations (35), (36), (37), and (38), respectively.

(Equation 34)

(Equation 35)

[Equation 36]

(37) In step S200, the end point ^V P _qi window regions 12 and 13 than those of formula, setting the ^V P _ri, ^V P _si, window regions 12 and 13 determine the ^V P _ti. It is a feature of the present invention to set the window regions 12 and 13 as viewed from the coordinate system on the vehicle coordinate system sigma _V That road surface as described above.

Here, the width d of the window regions 12 and 13
For example, by setting the width so that the lateral edge component of the vehicle parked in the adjacent parking lot is not detected, the lateral edge of the vehicle does not enter the window area, so the lateral edge of the vehicle is indicated by a white line. It will not be erroneously recognized as an edge.

Next, in step S210, the amount of movement of the vehicle is calculated based on the wheel angular velocity detected by the wheel speed sensor 2. Set the angular velocity of the left and right wheels to u
and _{l (t), u r (} t), the left and right wheel diameter _{R l, R r, P (} t 0) the position at time t = t _0, as described above when the tread and T = [0, 0,0,0] ^T
The position P (t ₀ + τ) at time t = t ₀ + τ is the second position (2
Since it is given by the expression (2), the expression (22) is repeated from t = t ₀ to t = t ₁ to obtain ^{V (t0)} PV _(t1) and Δθ
(T ₀ ; t ₁ ) is obtained. In step S220, a camera on which image processing is to be performed is selected based on the result of step S210 and the result obtained by the algorithm of FIG.

[0034] In step S230, the Camera C _j selected, the (29) or second (30) end point ^V P _qi window regions 12 and 13 from the equation, ^V P _ri, ^V P
_si, ^{Cj (t1)} for ^{_{V P ti r qi = [Cj}} (t1) X qi, Cj (t1)
Y _qi , ^{Cj (t1)} Z _qi ] ^T , ^{Cj (t1)} r _ri = [ ^{Cj (t1)} X _ri ,
^{Cj (t1)} _Yri , ^{Cj (t1)} _Zri ] ^T , ^{Cj (t1)} _rsi = [ ^{Cj (t1)} X
_si , ^{Cj (t1)} _Ysi , ^{Cj (t1)} _Zsi ] ^T , ^{Cj (t1)} _rti = [
^{Cj (t1)} _Xti , ^{Cj (t1)} _Yti , and ^{Cj (t1)} _Zti ] ^T are obtained and converted into a camera coordinate system. Next, in step S240,
From the expressions (11) and (12) or the expressions (13) and (14), the coordinates [ ^{Cj (t1)} _xqi , ^{Cj (t1)} _yqi ] ^T , [ ^{Cj (t1)} _xqi ] on the imaging screen of the camera Cj are obtained. ^{Cj (t1)} x _ri , ^{Cj (t1)}
y _ri ] ^T , [ ^{Cj (t1)} _xsi , ^{Cj (t1)} _ysi ] ^T ,
[ ^{Cj (t1)} _xti , ^{Cj (t1)} _yti ] ^T is obtained, and is converted into a shooting screen coordinate system. Window setting area 1 surrounded by these endpoints
Since 2A and 13A are trapezoids as shown in FIG. 14, in step S250, trapezoids surrounding the trapezoids are set in the window regions 12B and 13B as shown in FIG. Set.

In step S260, the window 12B,
After calculating the sum of the density values of all the pixels on the straight line to be detected in 13B and creating a table on the memory in the control device 1, in step S270, the maximum value in the table is detected. The straight line corresponding to the detected maximum value is the straight line having the highest accuracy as the edge straight line. Then, in step S280, tracking of pixels having a predetermined density value or more is performed, and the point where the pixel is interrupted is set as the end point of the white line of the garage frame. In step S290, the coordinates of the upper end of the end point on the camera screen are [ ^{Cj (t1)} _xui , ^{Cj (t1)} _yui ], and the coordinates of the lower end are [ ^{Cj (t1).}
_xli , ^{Cj (t1)} _yli ] (i = 1 to 2 ₎ , the position ^{V (t1) in} the vehicle coordinate system ΣV _(t1) of the end point of the white line according to the expressions (26) to (28 ^). P _ui [ ^{V (t1)} X _ui , ^{V (t1)} Y _ui , 0] ^T and
^{V (t1)} _Pli [ ^{V (t1)} _Xli , ^{V (t1)} _Yli , 0] ^T is obtained.

In step S300, the obtained ^{V (t1)} P _ui
[ ^{V (t1)} X _ui , ^{V (t1)} Y _ui , 0] ^T and ^{V (t1)} P _li [
^{Based on V (t1)} _Xli , ^{V (t1)} _Yli , 0] ^T , it is determined whether or not the vehicle A has reached the target point. If the vehicle A has not reached the target point, the process proceeds to step S310. In step S310, the vehicle A is driven by a predetermined amount along the target track. Then, returning to step S200, the same processing is repeated. The driving of the vehicle A in step S310 is driven by a driving device (not shown). If it is determined in step S300 that the target point has been reached, the process ends.

The above processing is summarized as follows. First, the vehicle is stopped before the garage frame drawn by the white line to be automatically parked. Next, the white line is imaged by the side camera, and the position of the white line is recognized. In this case, the point on the ground where the vehicle is currently stopped is set as the reference coordinates, and the positional relationship between the vehicle and the white line is accurately recognized on the reference coordinates. This includes
It is obtained by performing coordinate conversion based on the relationship between the photographing screen coordinate system and the camera coordinate system and between the camera coordinate system and the vehicle coordinate system. At this time, the reference coordinate system and the vehicle coordinate system match. Next, a predetermined range around the white line recognized in the reference coordinate system is set as a range corresponding to a window on which image processing is to be performed.

Next, when the vehicle moves along the automatic parking trajectory obtained by a predetermined method, or when the vehicle moves by the driver's operation, the amount of movement of the vehicle is obtained by the wheel speed sensor of the rear wheel. Based on the moving amount of the vehicle, it is possible to predict how the window set above moves on the photographing screen of the camera. This prediction is based on the relationship between the reference coordinate system based on the amount of vehicle movement and the vehicle coordinate system of the moved vehicle, the vehicle coordinate system and the camera coordinate system,
Prediction can be performed by performing coordinate conversion based on the relationship between the camera coordinate system and the shooting screen coordinate system. That is, after the vehicle moves, it is predicted how the window that seems to include the white line moves on the photographing screen of the camera.

In the window after the movement prediction, image processing is performed to extract a white line. When the white line is extracted,
From the coordinates of the white line on the shooting screen, coordinate conversion is performed based on the relationship between the shooting screen coordinate system and the camera coordinate system, the camera coordinate system and the vehicle coordinate system, the relationship between the vehicle coordinate system and the reference coordinate system, and the position of the vehicle and the white line in the reference coordinate system. Relationships can be accurately recognized. The moving amount of the vehicle obtained based on the wheel speed sensor is a so-called moving amount of the temporary vehicle, and the positional relationship between the white line and the vehicle obtained by this image processing becomes accurate moving data of the vehicle. By repeating such processing, automatic parking or driving by a driver can be performed while always recognizing the exact position of the host vehicle.

As described above, the setting of the window area to be subjected to the image processing is calculated and set based on the moving amount of the own vehicle, so that the position and the posture of the own vehicle are large with respect to the white line of the garage frame. Even if it changes, the white line can be reliably followed, and the exact position of the vehicle can always be recognized. Further, even when the white line once deviates from the field of view of the camera due to a large change in the position and attitude of the own vehicle, it is possible to reliably follow the white line again when it enters the field of view of the camera.

Further, when a plurality of different cameras are used, even if the white line deviates from the field of view of one camera, it can be easily determined which of the other cameras is in the field of view, and the white line can be reliably detected. Can follow. Further, when the trajectory of the vehicle is known in advance and a plurality of cameras are used, it is possible to calculate in advance which of the cameras is within the field of view of the object along the route along the trajectory, which is efficient. A lean image processing control strategy is established. In addition, since it is not necessary to track white lines from continuously captured images, usually, separate image processing is performed, and time sharing such as processing white lines at necessary places is possible, so image processing devices are limited. Resources can be used effectively.

Although the above embodiment has been described with reference to the example in which the wheel speed sensor is used to measure the amount of movement of the vehicle, the present invention is not limited to this. For example, a combination of a wheel speed sensor and a yaw rate sensor may be used, or another method may be used. In other words, any content may be used as long as the amount of movement including the rotation angle of the vehicle can be detected.

In the above-described embodiment, the extraction of the white line of the garage frame of the parking lot has been described as an example. However, the present invention is not limited to the white line on the parking lot or on the road. Any object can be used. For example, the present invention can be applied to a device such as a guardrail or a specific step.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of an embodiment of a vehicle image processing apparatus according to the present invention.

FIG. 2 is a plan view when the vehicle is viewed from above.

FIG. 3 is a view of the vehicle as viewed from behind.

FIG. 4 is a right side view of a rear portion of the vehicle.

5 is a diagram showing the relationship between the position of the object P viewed from the vehicle coordinate system sigma _V and the camera coordinate system sigma _C1.

6 is a diagram showing the relationship between the camera coordinate system and sigma _C shooting screen coordinate system.

FIG. 7 is a diagram illustrating a relationship between a vehicle coordinate system ΣV _(t0) before movement and a vehicle coordinate system ΣV _(t1) after movement.

FIG. 8 is a flowchart for making a global plan as to which of a plurality of cameras performs image processing at the start of automatic parking.

FIG. 9 is a flowchart for calculating a window setting area based on a measured vehicle movement amount.

FIG. 10 is a diagram showing a white line and an initial window captured by a camera.

FIG. 11 is a diagram showing an end point of a white line of an extracted garage frame captured by a camera.

12 is a diagram illustrating a white line and its end point in the vehicle coordinate system sigma _V.

FIG. 13 is a diagram showing a state of a window set in a vehicle coordinate system.

FIG. 14 is a diagram illustrating a state of a window set in a shooting screen coordinate system.

FIG. 15 is a diagram illustrating a window set in a trapezoidal shape in order to increase processing efficiency.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 control device 2 wheel speed sensor 3 image memory 4 white line processing unit 5 window coordinate calculation unit 6 vehicle movement amount calculation unit 7 vehicle coordinate calculation unit 8 trajectory planning unit 9 window setting unit 11 white line 12, 13 window A vehicle C1, C2 camera

Continued on the front page (51) Int.Cl. ⁶ Identification symbol FI H04N 7/18 H04N 7/18 J // G06T 7/20 G06F 15/70 410

Claims (7)

[Claims] 1. An image pickup device provided in a vehicle for picking up an image of an environment around the vehicle, and an image processing region setting for setting an image processing region for performing image processing in a part of an image picked up by the image pickup device. Means for extracting an object existing in the surrounding environment of the vehicle, the position of which changes relative to the vehicle as the vehicle moves, and the position of which does not change relative to the moving surface of the vehicle. An image processing device for performing image processing in an image processing area set by the area setting means, the vehicle image processing apparatus further comprising: a vehicle movement amount measurement means for measuring a movement amount of the vehicle on the moving surface; The image processing area setting means sets the image processing area after the movement of the vehicle based on the movement amount of the vehicle measured by the vehicle movement amount measurement means. Equipment. 2. The image processing apparatus for a vehicle according to claim 1, wherein the image processing area set after the movement of the vehicle is image-processed based on a positional relationship of an object extracted.
An image processing apparatus for a vehicle, which recognizes a position of a vehicle after moving. 3. An image processing apparatus for a vehicle according to claim 1, wherein said image pickup means comprises a plurality of image pickup apparatuses, and said image processing area setting means comprises a movement amount of the vehicle measured by said vehicle movement amount measurement means. An image processing apparatus for a vehicle, wherein one of the plurality of image capturing apparatuses for setting the image processing area after the vehicle is moved is selected based on the following. 4. The image processing apparatus for a vehicle according to claim 1, wherein a predetermined position on the moving surface when the vehicle is at a predetermined position on the moving surface is set as a reference coordinate, and an image pickup screen of the image pickup device is provided. A coordinate conversion means for mutually transforming the reference coordinates and the imaging screen coordinates in consideration of the movement amount of the vehicle from the predetermined position measured by the vehicle movement amount measurement means. The image processing apparatus for a vehicle, further comprising: the coordinate conversion unit converting the coordinates of the target object extracted on the imaging screen on the imaging screen coordinates into the reference coordinates. 5. The image processing apparatus for a vehicle according to claim 4, wherein the image processing area setting means sets an area including the extracted object on the reference coordinates, and thereafter, when the vehicle moves. An image processing apparatus for a vehicle, wherein an image processing area on the imaging screen is set by performing coordinate conversion of the area into the imaging screen coordinates by the coordinate conversion means. 6. An image pickup means provided in a vehicle for picking up an image of an environment around the vehicle, and an image processing area setting for setting an image processing area for performing image processing in a part of an image screen picked up by the image pickup means. Means for extracting an object existing in the surrounding environment of the vehicle, the position of which changes relative to the vehicle with the movement of the vehicle, and the position of which does not change relative to the moving surface of the vehicle. An image processing apparatus for a vehicle, comprising: an image processing unit configured to perform image processing in an image processing area set by an area setting unit; a vehicle trajectory setting unit configured to set a trajectory of a vehicle; A trajectory planning means for planning in advance how to set the image processing area based on vehicle position information at each point. 7. The image processing apparatus for a vehicle according to claim 6, wherein said image pickup means comprises a plurality of image pickup apparatuses, and the planning by said trajectory planning means determines which one of said plurality of image pickup apparatuses is to be selected. An image processing device for a vehicle, comprising a plan.