JP2009093332A - Vehicle peripheral image processor and vehicle peripheral circumstance presentation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To detect and notify an object existing in the peripheral air of a vehicle. <P>SOLUTION: This vehicle peripheral image processor is provided with a first image conversion part 13a for defining a horizontal face which is higher than the photographic position of a camera 2 as a reference face, and for creating a first bird's-eye view image projected on the reference face by converting the coordinates of image in a range higher than the photographic position of the camera 2; a second image conversion part 13b for creating a bird's-eye view image for display projected on the ground by converting the coordinates of the image; a featured point setting part 13c for setting a featured point in the bird's-eye view image; and a moving amount detection part 13d for detecting the moving amounts of a self-vehicle. When the self-vehicle moves, a solid determination part 13e determines and notifies whether or not the featured point is set in a solid existing at a position higher than the height of the photographic position of the camera 2 from the moving amounts of the self-vehicle and the change of the featured point in the first bird's-eye view image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a vehicle periphery image processing apparatus and a vehicle periphery state presentation method for processing an image obtained by capturing the periphery of a vehicle.

2. Description of the Related Art Conventionally, there has been known a vehicle periphery image providing device that captures a vehicle periphery with a plurality of cameras, generates an image that looks at the vehicle from a virtual viewpoint by performing coordinate conversion on the obtained image, and presents the image to the driver. Yes. Such a vehicle periphery image providing device generates a bird's-eye view image using the coordinate conversion reference plane as the ground, and presents the bird's-eye view image to the driver. As a result, the driver can objectively recognize the positional relationship between the white line on the ground and the own vehicle, and assists the parking driving and the margin driving. (See, for example, Patent Document 1 below).
JP 2003-169323 A

  However, in the conventional vehicle periphery image providing device, since the reference plane for creating the bird's-eye view image is the ground, objects existing in the air are not accurately represented on the bird's-eye view image. Therefore, it is difficult for the driver to distinguish between an object existing in the air and a white line or the like existing on the ground.

  Therefore, the present invention has been proposed in view of the above-described circumstances, and provides a vehicle periphery image processing apparatus and a vehicle periphery state presentation method that can detect and notify an object existing in the air around the vehicle. For the purpose.

  According to the present invention, when providing the driver with the situation around the vehicle, a vehicle periphery image obtained by capturing the periphery of the vehicle is generated by the imaging unit, and a horizontal plane higher than the height of the imaging position of the imaging unit is used as a reference plane. A bird's-eye view image data obtained by projecting coordinates on an image in a range higher than the imaging position is projected on the reference plane. Then, a feature point is set in the created overhead image data, and when the own vehicle moves, a movement amount of the own vehicle and a change in the overhead image data of the set feature point are obtained and set in the overhead image data. It is determined whether or not the feature point is set to a three-dimensional object that exists at a position higher than the height of the photographing position of the photographing means, and the driver is notified whether or not a three-dimensional object exists.

  According to the present invention, when the own vehicle moves, the actual amount of movement of the own vehicle and the change of the feature point in the overhead image data are obtained, and an object existing in the air around the vehicle can be detected and notified. it can.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  As shown in FIG. 1, for example, the present invention is applied to a vehicle periphery image providing device that provides a driver with an image around the vehicle. The vehicle surrounding image providing apparatus includes an image processing apparatus 1, a camera module (photographing means) 2 a, 2 b, 2 c, 2 d (hereinafter simply referred to as “camera module 2”), a vehicle speed sensor 3, and a steering. An angle sensor 4, a shift position sensor 5, a monitor (display means, notification means) 6, and a speaker (voice output means, notification means) 7 are connected.

  The camera module 2 captures the periphery of the vehicle, and for example, a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal-Oxide Semiconductor) camera is used. The camera module 2 includes four camera modules 2a, 2b, 2c, and 2d. Image data obtained by photographing by each camera module 2 is transmitted to the image processing apparatus 1. In addition, when displaying a bird's-eye view image (second bird's-eye view image data) as if the vehicle is viewed from above, it is desirable that four camera modules 2 exist, but an airborne object in one direction is detected as described later. In order to do so, it is sufficient that at least one camera module 2 is provided.

  The vehicle speed sensor 3 detects the vehicle speed of the host vehicle. A vehicle speed signal detected by the vehicle speed sensor 3 is supplied to the image processing apparatus 1. The steering angle sensor 4 detects the steering angle of the host vehicle. A steering angle signal detected by the steering angle sensor 4 is supplied to the image processing apparatus 1. The shift position sensor 5 detects the shift position (gear position) of the host vehicle. The shift position signal detected by the shift position sensor 5 is supplied to the image processing apparatus 1.

  The image processing apparatus 1 processes an image around the vehicle photographed by the camera module 2. The image processing apparatus 1 is configured by hardware by a computer, but in FIG. 1, for the sake of convenience, description is given separately for each functional block.

  The image processing apparatus 1 includes an input buffer 11a, 11b, 11c, 11d corresponding to each of the camera modules 2a, 2b, 2c, 2d (hereinafter referred to simply as “input buffer” when collectively referred to). 11 ”), a CPU 12, an image processing unit 13, a table storage unit 14, and an output buffer 15.

  The input buffer 11 receives and stores image data from the camera module 2. The input buffer 11 is provided corresponding to the number of camera modules 2, the input buffer 11a is connected to the camera module 2a, the input buffer 11b is connected to the camera module 2b, and the input buffer 11c is connected to the camera module 2c. The input buffer 11d is connected to the camera module 2d. The input buffer 11 temporarily stores the image data, and the image data is read at the image conversion timing by the image processing unit 13 under the control of the CPU 12.

  The output buffer 15 stores the overhead image for display (second overhead image data) whose coordinates have been converted by the image processing unit 13 described later. The output buffer 15 outputs the stored overhead image information to the monitor 6.

  The monitor 6 displays the overhead image output from the output buffer 15. In addition, when the image processing unit 13 determines that the feature point is set as a three-dimensional object as described later, the monitor 6 highlights the three-dimensional object. The monitor 6 may highlight the three-dimensional object in the overhead view image based on the detected actual distance between the vehicle end (camera mounting position) and the aerial object. As a highlighting method, an information image that informs the driver that the object on the overhead image is a three-dimensional object is displayed superimposed on the overhead image.

  The speaker 7 is configured to notify the driver of the presence of the three-dimensional object with a predetermined sound when it is determined by the image processing unit 13 described later that the feature point is set to the three-dimensional object. Specifically, the speaker 7 informs the driver of the presence of the three-dimensional object with a beep sound such as “beep, beep, beep” or a conversation sound such as “There is an obstacle on the left side”.

  The CPU 12 controls the entire image processing apparatus 1. The CPU 12 mainly controls the timing at which image data is extracted from the input buffers 11a, 11b, 11c, and 11d, the image processing timing of the image processing unit 13, and the timing at which the overhead image is transmitted from the output buffer 15 to the monitor 6. The CPU 12 includes devices such as an ASIC (Application Specific Integrated Circuit), an LSI (Large Scale Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

  The table storage unit 14 stores an address conversion table for coordinate conversion of image data around the vehicle photographed by the camera module 2. The table storage unit 14 includes, as an address conversion table, a first address conversion table 14a for performing coordinate conversion of image data using a horizontal plane higher than the height of the shooting position of the camera module 2 as a reference plane, and an image using the ground as a reference plane. A second address conversion table 14b for converting data coordinates is stored.

  The image processing unit 13 performs first and second image conversion units (second image conversion unit) 13b and a first image conversion unit (first image conversion unit) 13b that perform coordinate conversion using image data to create an overhead image. A feature point setting unit (feature point setting unit) 13c for setting a feature point on the coordinate-converted overhead image, a movement amount detection unit (movement amount detection unit) 13d for detecting the movement amount of the vehicle, and the feature point A solid determination unit (a solid determination unit) 13e that determines whether the solid object is set.

  The first image conversion unit 13a refers to the first address conversion table 14a stored in the table storage unit 14 and performs coordinate conversion of the image data. The first image conversion unit 13a uses image data in a range higher than the imaging position (attachment position) of the camera module 2 among the image data obtained from the input buffer 11, and is higher than the height of the imaging position of the camera module 2. Coordinate conversion is performed using a high horizontal plane as a reference plane, and first overhead image data for detecting a three-dimensional object higher than the imaging position of the camera module 2 is created. This first bird's-eye view image data is a bird's-eye view image in which an upper reference plane is viewed from a position lower than the imaging position of the camera module 2.

  The second image conversion unit 13b refers to the second address conversion table 14b stored in the table storage unit 14 and performs coordinate conversion of the image data. This second image conversion unit 13b uses image data in a range lower than the imaging position (attachment position) of the camera module 2 among the image data obtained from the input buffer 11, and performs coordinate conversion using the ground as a reference plane. Second display image data for display is created. The second bird's-eye view image data is a bird's-eye view image as if the host vehicle was seen from above, and is stored in the output buffer 15 by the image processing unit 13 and then output to the monitor 6.

  The feature point setting unit 13c sets the first feature point in the first overhead image data created by the first image conversion unit 13a. The feature point setting unit 13c also sets the second feature point in the second overhead image data created by the second image conversion unit 13b. The feature point setting unit 13c first detects color information or brightness information of each pixel constituting the first and second overhead image data, and detects an edge included in the first and second overhead image data. Then, the edge line segment in which the edge is arranged is detected. Next, the feature point setting unit 13c sets a feature point in the detected edge line segment or a region surrounded by the edge line segment.

  Based on the vehicle speed signal output from the vehicle speed sensor 3, the steering angle signal output from the steering angle sensor 4, and the shift position signal output from the shift position sensor 5, the movement amount detection unit 13 d The amount of movement is detected.

  When the movement detection unit 13d detects the movement of the host vehicle, the three-dimensional determination unit 13e detects the movement amount of the host vehicle detected by the movement amount detection unit 13d and the first set by the feature point setting unit 13c. From the change in the first overhead image data of the feature point, it is determined whether or not the first feature point is set to a three-dimensional object that exists at a position higher than the shooting position of the camera module 2. Further, when the movement of the host vehicle is detected by the movement amount detection unit 13d, the three-dimensional determination unit 13e is set by the movement amount of the own vehicle detected by the movement amount detection unit 13d and the feature point setting unit 13c. It is determined from the change in the second overhead image data of the second feature point whether or not the second feature point is set as a three-dimensional object on the ground.

  When a plurality of first feature points are set, the three-dimensional determination unit 13e compares the movement amount of the host vehicle with the movement amounts of all the first feature points on the first overhead image data. It is determined whether one feature point is set on the three-dimensional object. In addition, when a plurality of second feature points are set, the solid determination unit 13e compares the movement amount of the host vehicle with the movement amounts of all the second feature points on the second overhead image data, It is determined whether each second feature point is set on the three-dimensional object.

  When the first feature point and the second feature point are present on the three-dimensional object, the vehicle periphery image providing device indicates that the object in which the first feature point and the second feature point are set is a three-dimensional object. In order to notify the driver, an image process for highlighting the object and a voice notification of the presence of the object are performed.

  Processing of the second image conversion unit 13b that creates such second overhead image data will be described with reference to FIG. The process of the second image conversion unit 13b is a case where there is no solid object around the host vehicle.

  The coordinate conversion processing for creating the overhead image 101 by the second image conversion unit 13b includes a vehicle peripheral image 102a in front of the vehicle imaged by the camera module 2a, a vehicle peripheral image 102b in the vehicle rear imaged by the camera module 2b, A vehicle peripheral image 102c on the left side of the vehicle photographed by the camera module 2c and a vehicle peripheral image 102d on the right side of the vehicle photographed by the camera module 2d are used. In the vehicle peripheral images 102a to 102d and partial images 104a to 104d described later shown in FIG. 2, a white line 103 drawn on the ground around the vehicle is displayed.

  The second image conversion unit 13b performs coordinate conversion of each of the vehicle surrounding images 102a to 102d with reference to the second address conversion table 14b. That is, the second image conversion unit 13b generates a partial image 104a in which the vehicle periphery image 102a in front of the vehicle is coordinate-converted to view the front of the host vehicle from the sky, and the vehicle periphery image 102b in the rear of the vehicle is coordinate-converted to generate the partial image 104a. A partial image 104b in which the rear of the vehicle is viewed from above is generated, and a vehicle peripheral image 102c on the left side of the vehicle is coordinate-converted to generate a partial image 104c in which the left side of the vehicle is viewed from above. Is transformed to generate a partial image 104d in which the right side of the vehicle is viewed from above. Then, the second image conversion unit 13b synthesizes the partial images 104a to 104d obtained by the coordinate conversion to generate a display bird's-eye view image 101 in which the periphery of the host vehicle is viewed from above. The second image conversion unit 13b arranges (imposes) a mark 105 indicating the host vehicle in the center of the overhead image 101 in order to present the positional relationship between the vehicle periphery and the host vehicle to the driver.

  As described above, the second image conversion unit 13 b performs coordinate conversion on the vehicle periphery image captured by the camera module 2 to generate the overhead image 101, and outputs the overhead image 101 to the output buffer 15.

  Similarly to the second image conversion unit 13b, the first image conversion unit 13a also captures image data captured by the camera modules 2a, 2b, 2c, and 2d and stored in the input buffers 11a, 11b, 11c, and 11d. Is transformed. The first bird's-eye view image data created by the first image conversion unit 13a is an image obtained by viewing the reference plane at a position higher than the imaging position of the camera module 2 from below the imaging position of the camera module 2. The first bird's-eye view image data is temporarily stored in a memory (not shown) of the image processing unit 13 without being displayed on the monitor 6.

  Next, the relationship between the imaging position of the camera module 2, the position of the three-dimensional object, and the state of the overhead view image when a three-dimensional object exists on the ground or in the air around the host vehicle will be described with reference to FIG. To do. FIG. 3 shows the positional relationship on the overhead image between the image 301 ′ of the host vehicle representing the host vehicle 301 and the images 303A ′ and 303B ′ representing the three-dimensional obstacles 303A and 303B in the upper and lower stages. Yes.

  As shown in FIG. 3, when a three-dimensional obstacle 303 </ b> A having a height h exists on the ground behind the host vehicle 301, the camera module 2 starts imaging from the imaging position 302 (= installation position) having a height H. Consider a case where a three-dimensional obstacle 303A having a height h existing at a position lower than the position 302 is imaged. In this case, the imaging line L1 of the camera module 2 intersects the upper end portion of the three-dimensional obstacle 303A, and the imaging line L2 of the camera module 2 intersects the ground point of the three-dimensional obstacle 303A. In this way, when a video obtained by imaging the three-dimensional obstacle 303A from the imaging position 302 of the camera module 2 is coordinate-converted using the ground as a reference plane to create an overhead image, as shown in the lower part of FIG. The image 303A ′ is drawn. In this bird's-eye view image (second bird's-eye view image data), a three-dimensional obstacle image 303A ′ representing a grounding point of the three-dimensional obstacle 303A is drawn at a position P1 at a distance a from the own vehicle 301, and a distance b from the own vehicle 301 is drawn. As the position P2, an image 303A ′ of the three-dimensional obstacle 303 representing the upper end portion of the three-dimensional obstacle 303A is drawn.

  When a three-dimensional obstacle 303B having a height h exists in the air behind the host vehicle 301, the camera module 2 is positioned higher than the imaging position 302 from the imaging position 302 (= installation position) having the height H. Let us consider a case in which a three-dimensional obstacle 303B having a height h existing in FIG. In this case, the imaging line L1 'of the camera module 2 intersects with the lower end portion of the stereoscopic obstacle 303A, and the imaging line L2' of the camera module 2 intersects with the upper end point of the stereoscopic obstacle 303B. In this way, an image obtained by imaging the three-dimensional obstacle 303B from the imaging position 302 of the camera module 2 is referred to the aerial horizontal reference plane 304 that is higher than the imaging position 302 of the camera module 2 and contacts the upper end of the three-dimensional obstacle 303B. As shown in the upper part of FIG. 3, a three-dimensional obstacle image 303B ′ is drawn. In this bird's-eye view image (first bird's-eye view image data), an image 303B ′ representing the upper end portion of the three-dimensional obstacle 303B is drawn as a position P1 at a distance a from the host vehicle 301, and as a position P2 at a distance b from the host vehicle 301. An image 303B ′ representing the lower end of the three-dimensional obstacle 303B is drawn.

  As described above, when the imaging position 302 and the three-dimensional obstacles 303A and 303B are different in height, and when the three-dimensional obstacles 303A and 303B having a three-dimensional height h are imaged, a reference plane for creating an overhead image is obtained. The portions of the three-dimensional obstacles 303A and 303B that are in contact with each other are drawn as a position P1 at a distance a from the actual host vehicle 301. However, the portions of the three-dimensional obstacles 303A and 303B that are separated from the reference plane are drawn in a form in which the three-dimensional obstacles 303A and 303B are collapsed, unlike the actual distance from the host vehicle 301.

  Therefore, when the obstacle is a three-dimensional object, the vehicle periphery image providing device presents the fact to the driver by performing the processing described later.

  As described above, an overhead image to be presented to the driver when a three-dimensional object exists around the host vehicle will be described. The display bird's-eye view image presented to the driver is the bird's-eye view image (second bird's-eye view image data) obtained by performing coordinate conversion with the ground as a reference plane in FIG.

  When a three-dimensional obstacle and other vehicles exist around the host vehicle, for example, as shown in FIG. 4, the overhead image 401 displayed by the vehicle periphery image providing device includes a white line image around the host vehicle image 402. 403, pole image 404, and other vehicle image 405 are included. Here, the white line image 403 on the ground, which is the reference plane for coordinate conversion, is drawn at a correct position relative to the host vehicle 402 on the overhead image 401. However, as described above, the pole image 404 and the other vehicle image 405 that exist three-dimensionally from the reference surface that is the ground and are at a high position, that is, the three-dimensional object image, fall on the ground on the overhead image 401. It is drawn as if it were. Therefore, even if the driver looks at the bird's-eye view image 401, the driver feels that it is difficult to grasp the sense of distance from the host vehicle to the three-dimensional object. In addition, since the three-dimensional object looks like a thing on the ground, it is difficult to distinguish from a white line or the like actually on the ground.

  Therefore, the vehicle periphery image providing device determines whether the object included in the second overhead image data is a three-dimensional object by the feature point setting unit 13c, the movement amount detection unit 13d, and the three-dimensional determination unit 13e of the image processing unit 13. .

  When the overhead detection image 401 shown in FIG. 4 is subjected to edge detection processing by the feature point setting unit 13c, an edge image 501 as shown in FIG. 5 can be obtained. The edge image 501 includes a white line edge 503 obtained from the white line image 403, a pole edge 504 obtained from the pole image 404, and another vehicle edge 505 obtained from the other vehicle image 405 around the own vehicle image 502. Can be detected. Then, the feature point setting unit 13c sets a feature point on the edge line segment or in a region surrounded by the edge line segment. That is, the feature point setting unit 13c sets feature points on the edge portion or inside of the white line edge 503, the pole edge 504, and the other vehicle edge 505. Note that it is desirable that the feature point setting unit 13c sets not only one feature point but also a plurality of feature points, thereby setting feature points for all portions that may be solid objects.

  Then, the image processing unit 13 determines whether or not the feature point set on the second overhead image data is set on the three-dimensional object by the three-dimensional determination unit 13e. The movement amount detection unit 13d obtains a change in the movement amount in the second overhead image data of the feature point set by the feature point setting unit 13c when the host vehicle moves. Then, the solid determination unit 13e compares this movement amount with the movement amount of the host vehicle detected by the movement amount detection function, and determines whether or not the feature point is set on the three-dimensional object.

  The operation of the three-dimensional determination unit 13e at this time will be described with reference to FIGS. 6 to 8 illustrate the positional relationship between the image 301 ′ representing the host vehicle 301 and the image 303 ′ representing the three-dimensional obstacle 303 on the overhead view.

  FIG. 6 shows the relationship between the host vehicle 301 and the three-dimensional obstacle 303A when the time t is t0, and FIG. 7 shows the relationship between the host vehicle 301 and the three-dimensional obstacle 303A when the time t is t1 (> t0). FIG. 8 is a diagram showing a change from time t = t0 to t = t1.

  In the situation at time t = t0 as shown in FIG. 6, there is a three-dimensional obstacle 303 (t0) having a height h from the ground behind the host vehicle 301, and the camera module 2 has an imaging position 302 having a height H. The three-dimensional obstacle 303 (t0) existing at a position lower than the imaging position 302 is imaged from (= installation position). In this case, the imaging line L1 (t0) of the camera module 2 intersects the upper end (height h) of the three-dimensional obstacle 303 (t0), and the imaging line L2 (t0) of the camera module 2 is the three-dimensional obstacle 303 (t0). Intersects the lower end of In this way, when an image obtained by imaging the three-dimensional obstacle 303 (t0) from the imaging position 302 of the camera module 2 is coordinate-converted using the ground as a reference plane to create a bird's-eye view image, as shown in the lower part of FIG. An obstacle image 303 ′ (t0) is drawn. In this bird's-eye view image (second bird's-eye view image data), the lower end of the three-dimensional obstacle 303 (t0) is drawn as a position P1 at a distance N farther than the actual distance R (t0), and a three-dimensional obstacle 303 ( An object video 303 ′ (t0) in which the upper end portion of t0) is drawn is drawn.

  Thereafter, when time t = t1, as shown in FIG. 7, the host vehicle 301 moves backward and the actual distance from the three-dimensional obstacle 303 (t1) approaches R (t1) from time t = t0. In this case, in the overhead view image, the lower end portion of the three-dimensional obstacle 303 (t1) is drawn as the position P1 of the distance M farther than the actual distance R (t1), and the upper end portion of the three-dimensional obstacle 303 (t1) as the position P2. The object image 303 ′ (t1) in which is drawn is drawn.

  As shown in FIG. 8, when the moving distance S of the host vehicle 301 over time t0 to t1 is compared with the moving distance L (= N−M) of the three-dimensional obstacle 303 in the air on the overhead view image, the actual movement The movement distance L on the bird's-eye view image is longer than the distance S, and there is a difference between the actual movement amount and the movement amount on the bird's-eye view image. This is because the object on the ground moves by the amount of movement of the host vehicle 301. However, on the principle of the overhead view image, the position of the lower end of the three-dimensional obstacle 303 in the air on the overhead view image is the actual position of the own vehicle 301. It moves more than the amount of movement.

  Therefore, as shown in FIG. 8, the three-dimensional determination unit 13e determines that the feature point is in the air based on whether the movement amount of the feature point set on the overhead image is larger than the actual movement amount of the host vehicle 301. It is possible to determine whether or not the feature point is set to a three-dimensional object.

  When the solid determination unit 13e determines that the feature point is set to a solid object, the imaging position H of the camera module 2 stored in advance, the movement distance S obtained by the movement amount detection unit 13d, and the overhead view image are stored. The height h of the three-dimensional obstacle 303 including the feature point is obtained by performing the following equation using the movement distance L of the feature point at.

h = H (1-S / L)
In addition, the solid determination unit 13e uses the actual movement distance S, the distance M to the three-dimensional obstacle 303 (t1) after movement in the overhead image, and the movement distance L of the feature point in the overhead image to calculate the following formula: Is performed to obtain the actual distance R between the rear end of the host vehicle 301 (the imaging position of the camera module 2) and the three-dimensional obstacle 303 in the air.

R = S × M / L
Although the actual distance R between the imaging position of the camera module 2 and the lower end of the three-dimensional obstacle 303 is obtained as described above, the distance can be similarly obtained even at the upper end of the three-dimensional obstacle 303. .

  As described above, according to this vehicle periphery image providing device, it is possible to detect that the object included in the overhead image created with the reference plane as the ground is a three-dimensional object.

  In addition, as described with reference to FIG. 3, the vehicle periphery image providing device uses a horizontal plane higher than the height of the shooting position of the camera module 2 as a reference plane and a partial image in a range higher than the imaging position of the camera module 2. The coordinates of 603 are transformed to create first bird's-eye image data projected on the reference plane. For example, when the camera module 2 captures image data 601 including a partial image 602 on the ground and a partial image 603 in the air as shown in FIG. 9 by the camera module 2, the partial image 603 in the air is used. Second overhead image data is created. This aerial partial image 603 is a predetermined one of the image data captured by the camera module 2 in advance based on the imaging position of the camera module 2, the imaging direction (mounting direction) of the camera module 2, and distortion (distortion aberration) data. It is determined to be a range. Therefore, the first image conversion unit 13a can obtain an aerial partial image 603 by extracting image data in a predetermined range from the image data extracted from the input buffer 11.

  For example, when the own vehicle enters a parking lot with a roof such as a garage, the aerial partial image 603 includes an obstacle such as a roof, and the second overhead image data is drawn with an aerial obstacle. It will be. Then, the vehicle periphery image providing device performs edge detection processing on the second overhead image data, and sets a feature point (second feature point) on the edge line segment or in an area surrounded by the edge line segment. From the amount of movement of the host vehicle and the change in the feature point in the second bird's-eye view image data, it is determined whether the feature point is set as an airborne three-dimensional object.

  Next, as described with reference to FIGS. 6 to 8, the overhead image changes when the host vehicle 301 moves backward toward the three-dimensional obstacle 303 in the air over time t = t0 to t = t1. Will be described with reference to FIGS.

  As shown in FIGS. 10 to 12, the horizontal reference plane 704 in the air is set to a position that is higher than the imaging position 702 of the camera module 2 by a height H. The vehicle periphery image providing device detects a three-dimensional obstacle 703 that exists in a range from a position higher than the height of the photographing position of the camera module 2 to a position lower than the horizontal reference plane 704 in the air.

  In the situation at time t = t0 as shown in FIG. 10, the three-dimensional obstacle 703 (t1) is located above the ground behind the host vehicle 701 by a height h ′ and below the aerial horizontal reference plane 704 by a height h. Is present, the camera module 2 images the three-dimensional obstacle 703 that exists in a range higher than the imaging position 702 from the imaging position 402 (= installation position). In this case, the imaging line L1 (t0) of the camera module 2 intersects the lower end of the three-dimensional obstacle 703 (t0), and the imaging line L2 (t0) of the camera module 2 intersects with the upper end of the three-dimensional obstacle 703 (t0). Intersect. In this way, when the video obtained by imaging the three-dimensional obstacle 703 (t0) from the imaging position 702 of the camera module 2 is coordinate-converted with reference to the horizontal reference plane 704 in the air, an overhead image is generated as shown in FIG. A three-dimensional obstacle image 703 ′ as shown in the upper part of the middle is included. In this bird's-eye view image (first bird's-eye view image data), the upper end of the three-dimensional obstacle 703 (t0) is drawn as a position P2 at a distance N far from the actual distance R (t0), and a three-dimensional obstacle 703 ( An object video 703 ′ (t0) in which the lower end of t0) is drawn is drawn.

  Thereafter, when time t = t1, as shown in FIG. 11, the own vehicle 701 moves backward and the actual distance from the three-dimensional obstacle 703 (t1) approaches R (t1) from time t = t0. In this case, in the overhead image, the upper end of the three-dimensional obstacle 703 (t1) is drawn as the position P1 at a distance M farther than the actual distance R (t1), and the lower end of the three-dimensional obstacle 703 (t1) as the position P2. The object image 703 ′ (t1) in which is drawn is drawn.

  In the description of FIGS. 10 to 12, the calculation is performed using the upper end of the three-dimensional obstacle 703 in the air as the first feature point. However, the distance R is measured by setting the first feature point at the lower end of the three-dimensional obstacle 703. You may do it.

  As shown in FIG. 12, when the moving distance S of the host vehicle 701 is compared with the moving distance L (= N−M) of the three-dimensional obstacle 703 in the air on the overhead image, the overhead image is larger than the actual moving distance S. The moving distance L on the upper side becomes longer, and there is a difference between the actual moving amount and the moving amount on the overhead image. This is because the object on the ground moves by the amount of movement of the host vehicle 701, but on the principle of the overhead view image, the position of the upper end of the three-dimensional obstacle 703 in the air on the overhead view image is the actual position of the own vehicle 701. It moves more than the amount of movement.

  Therefore, as shown in FIG. 12, the solid determination unit 13e determines that the feature point is in the air based on whether the movement amount of the feature point set on the overhead image is larger than the actual movement amount of the host vehicle 701. It can be determined whether or not it is set below the horizontal reference plane 704, that is, whether or not the feature point is set to a three-dimensional object.

  At this time, the solid determination unit 13e is obtained by the height (= H) between the horizontal reference plane 704 in the air and the imaging position of the camera module 2 and the movement amount detection unit 13d, as in the case where the ground is the reference plane. The height h ′ from the ground of the three-dimensional obstacle 703 including the feature point can be obtained by calculating the following expression using the movement distance S and the movement distance L of the feature point in the overhead image.

h ′ = 2H−H (1−S / L) = H (1 + S / L)
If the horizontal reference plane 704 in the air is not set to the height H from the imaging position 702 of the camera module 2 and is set to the height H ′ from the imaging position 702 of the camera module 2, a three-dimensional obstacle The height h ′ from the ground of 703 is
h ′ = (H + H ′) − H (1−S / L) = H (1 + S / L)
It becomes. Therefore, for example, when it is desired to detect a three-dimensional obstacle 703 that may cause the host vehicle 701 to collide when the host vehicle 701 moves backward, the height H ′ of the white line image 703 can be set at the height of the host vehicle 701. It is set to a height (H + H ′) from the ground that is approximately the same as the height of the host vehicle, to the extent that the allowance is added.

  Further, the solid determination unit 13e uses the actual movement distance S, the distance M to the three-dimensional obstacle 703 (t1) after movement in the overhead image, and the movement distance L of the feature point in the overhead image to calculate the following formula: , The actual distance R between the rear end of the host vehicle 701 (the imaging position of the camera module 2) and the three-dimensional obstacle 703 in the air is obtained.

R = S × M / L
As described above, according to this vehicle periphery image providing device, as shown in FIG. 13A, a three-dimensional object that exists three-dimensionally from the ground in a range 810 below the horizontal plane that includes the imaging position 802 of the camera module 2. The obstacle 803A can be detected by the second image conversion unit 13b creating second overhead image data and the feature point setting unit 13c, the movement amount detection unit 13d, and the three-dimensional determination unit 13e. However, for the three-dimensional obstacle 803B having the height portion 803a exceeding the imaging position 302 of the camera module 2, it is not possible to detect the height portion 803a. Therefore, according to this vehicle periphery image providing device, as shown in FIG. 13B, there is a case where there is a three-dimensional obstacle 803B that exists in a range 811 that is higher than the horizontal plane including the imaging position 802 of the camera module 2. In addition, the first image conversion unit 13a can create the first bird's-eye view image data, and the feature point setting unit 13c, the movement amount detection unit 13d, and the three-dimensional determination unit 13e can detect the entire three-dimensional obstacle 803B.

  In addition, as shown in FIGS. 10 to 12, the vehicle periphery image providing apparatus sets the height of the horizontal reference plane 704 in the air to twice the height H of the imaging position 702 of the camera module 2. A conversion formula (R = S × M / L) for measuring the distance R from the host vehicle 701 to the three-dimensional obstacle 703, and a conversion formula (R = S × M / L) for measuring the distance R to the three-dimensional obstacle 703 on the ground. = S × M / L).

  Next, a processing procedure for detecting a three-dimensional object existing around the host vehicle by the vehicle surrounding image providing apparatus configured as described above will be described with reference to a flowchart of FIG. Note that the processing shown in FIG. 14 starts when a bird's-eye view image for display is created from an image photographed by the camera module 2 and stored in the output buffer 15.

  First, in step S <b> 1, the image processing apparatus 1 determines whether the host vehicle has moved based on the vehicle speed signal output from the vehicle speed sensor 3. If it is determined that the host vehicle has not moved, this process is repeated until it is determined that the vehicle has moved. On the other hand, if it is determined that the host vehicle has moved, the process proceeds to step S2.

  In the next step S2, the behavior of the host vehicle is monitored by the movement amount detection unit 13d of the image processing unit 13. At this time, the movement amount detection unit 13d detects the movement amount and movement direction of the host vehicle as needed. In this process, the image processing unit 13 can detect information such as “the vehicle moves s [mm] forward”.

  In the next step S3, first overhead image data is created by the first image conversion unit 13a, and second overhead image data is created by the second image conversion unit 13b. At this time, the image processing unit 13 takes out the image data of each direction captured by the camera modules 2a, 2b, 2c, and 2d from the input buffers 11a, 11b, 11c, and 11d, and the first image conversion unit 13a Coordinate conversion is performed using the horizontal reference plane as a reference, and the second image conversion unit 13b performs coordinate conversion using the ground as a reference. The second overhead image data created by the second image conversion unit 13b is temporarily stored in the output buffer 15 and then displayed on the monitor 6.

  In the next step S4, the image processing unit 13 performs edge detection on the first overhead image data and the second overhead image data created in step S3. In the edge detection process, the color signals (RGB), color difference signals (Cb, Cr), or luminance signals (Y) of adjacent pixels on the first overhead image data and the second overhead image data are compared, and the predetermined value or more. If there is a difference between the pixels, the pixel is determined to be an edge. For example, as illustrated in FIG. 13, when there is no three-dimensional object in the aerial partial image of the image data, such as when there is no object higher than the imaging position 802 of the camera module 2, the second No feature points are set.

  Then, the image processing apparatus 1 sets the first feature point in the first overhead image data and the second feature point in the second overhead image data by the feature point setting unit 13c based on the result of the edge detection. Set.

  In the next step S5, the movement of the host vehicle detected by the movement amount detection unit 13d by the three-dimensional determination unit 13e and the first feature point and the second feature point set based on the edge detected in step S4. It is determined whether or not the movement matches. Here, if it is determined that the movement of the host vehicle and the movement of the second feature point set based on the edge match, it is determined that the second feature point is set on the ground that is not a three-dimensional object. To do. If the movement of the host vehicle does not match the movement of the second feature point, it is determined that the second feature point is set for the three-dimensional object on the ground, and the process proceeds to step S6. Furthermore, if the movement of the host vehicle and the movement of the first feature point do not match, the process proceeds to step S6 because an obstacle exists above the imaging position of the camera module 2.

  More specifically, for example, when the vehicle moves forward by a distance corresponding to three pixels in terms of a bird's-eye view image, the feature points set in the lines such as the white line are the same as the movement amount of the host vehicle. Is moved by a distance corresponding to three pixels. On the other hand, when another feature point moves by a distance corresponding to six pixels, the solid determination unit 13e determines that the object for which the feature point is set is not a white line or the like but a solid object. Furthermore, the image processing apparatus 1 sets the height h of the three-dimensional object in which the second feature point is set, the distance R from the vehicle end (the imaging position of the camera module 2) to the three-dimensional object, and the first feature point. The height h of the three-dimensional object and the distance R from the vehicle end (imaging position of the camera module 2) to the three-dimensional object are calculated.

  In the next step S6, the image processing unit 13 superimposes the information image for emphasizing the three-dimensional object image in the second bird's-eye image data determined as the three-dimensional object in step S5 on the second bird's-eye image data. At this time, the image processing unit 13 may perform highlighting according to the detected height h or h ′ of the three-dimensional object and the distance R to the three-dimensional object. For example, the monitor 6 causes the driver to perform highlighting as the height is such that the own vehicle is in contact with the vehicle or the distance from the own vehicle is closer. In addition, the vehicle periphery image providing device notifies the driver of the presence of the three-dimensional object with a predetermined sound through the speaker 7.

  Note that the three-dimensional object below the imaging position of the camera module 2 is included in the second overhead image data displayed on the monitor 6, but the three-dimensional object exists at a position higher than the imaging position of the camera module 2. An object may not be included in the second overhead image data. In this case, the image processing apparatus 1 may highlight the relative position of the three-dimensional object detected from the first bird's-eye image data with respect to the host vehicle using the second bird's-eye image data.

  Thereafter, the vehicle surrounding image providing device determines whether or not the ignition switch of the host vehicle is turned off, and if it is determined that the ignition switch is not turned off, the process proceeds to step S1 and the ignition switch is turned off. If it is determined, the process is terminated.

  As described above in detail, according to the vehicle periphery image providing device to which the present invention is applied, the second overhead image data obtained by performing coordinate conversion using the horizontal plane higher than the height of the shooting position of the camera module 2 as the reference plane is created. A first feature point is set in the first bird's-eye view image data, and the first feature point is determined from the imaging position of the camera module 2 based on the amount of movement of the host vehicle and the change of the first feature point in the first bird's-eye view image data. It is possible to determine whether or not the object is set to a high three-dimensional object in the air. Therefore, according to this vehicle periphery image providing device, it is possible to determine the presence or absence of a three-dimensional object present at a position higher than the imaging position of the camera module 2.

  Further, according to this vehicle periphery image providing device, the first feature point is higher than the imaging position of the camera module 2 from the movement amount of the host vehicle and the movement amount of the first feature point in the first overhead image data. Since it is determined whether or not the three-dimensional object is set in the air, the first feature point is set to the three-dimensional object from the difference between the actual movement amount of the own vehicle and the movement amount of the three-dimensional object in the overhead image. Can be determined.

  Furthermore, according to this vehicle periphery image providing device, when it is determined that the first feature point is set to a three-dimensional object in the air higher than the imaging position of the camera module 2, the imaging position of the camera module 2 is high. Since the height of the first feature point can be obtained from the movement amount of the host vehicle and the movement amount of the first feature point in the first bird's-eye view image data, display or sound according to the height of the three-dimensional object Output can be done.

  Furthermore, according to this vehicle periphery image providing device, when it is determined that the first feature point is set to a three-dimensional object in the air higher than the imaging position of the camera module 2, the amount of movement of the host vehicle, The actual distance between the vehicle and the three-dimensional object is obtained from the movement amount of the first feature point in the first overhead image data and the distance in the first overhead image data between the first feature point and the imaging position of the camera module 2. Therefore, it is possible to perform display or audio output for notifying a three-dimensional object according to the actual distance.

  Furthermore, according to this vehicle periphery image providing device, the first bird's-eye view image data is created by setting the height of the horizontal reference plane 704 in the air to be substantially the same as the height of the own vehicle. The presence / absence of a three-dimensional object existing in the air up to the horizontal reference plane 704 can be determined, and it can be notified that there is a three-dimensional object that the host vehicle may contact.

  Furthermore, according to this vehicle periphery image providing device, it is possible to display around the vehicle by the second overhead image data coordinate-converted with the ground as a reference plane, and to determine that a three-dimensional object exists on the ground. It can be displayed that the three-dimensional object in the second bird's-eye view image data with the ground as a reference plane is displayed as a form collapsed by coordinate conversion. Further, according to this vehicle periphery image providing device, in a wide range such as a range of height from the ground to the imaging position of the camera module 2 and a range of height from the imaging position of the camera module 2 to the horizontal reference plane 704 in the air. The presence or absence of a three-dimensional object can be determined.

  Furthermore, according to this vehicle periphery image providing device, the first horizontal bird's-eye image data is created by setting the horizontal reference plane 704 in the air to a height twice as high as the imaging position of the camera module 2. The distance between the three-dimensional object existing at a position higher than the imaging position of the camera module 2 and the own vehicle can be obtained by the same calculation as the calculation for obtaining the distance from the own vehicle, and the processing load can be reduced.

  Furthermore, according to this vehicle periphery image providing device, the edge line segment in the overhead image data is detected from the color information or the brightness information of each pixel on the first and second overhead image data, and the detected edge line segment is detected. Since the first and second feature points are set in the area surrounded by the upper or detected edge line segment, the first and second feature points are set on the edge or inside of a solid line or a solid object drawn on the ground. Will be. Therefore, the first and second feature points can be reliably set at a portion that may be a three-dimensional object.

  Furthermore, according to this vehicle periphery image providing device, when the presence of a three-dimensional object is determined from the first overhead image data, the three-dimensional object is highlighted, so that the three-dimensional object can be presented to the driver in an easily understandable manner. Further, even when the three-dimensional object of the first overhead image data is not displayed in the displayed second overhead image data, the three-dimensional object of the first overhead image data is notified to the driver in an easily understandable manner. Can do.

  Furthermore, according to this vehicle periphery image providing device, when it is determined that the first feature point or the second feature point is set as a three-dimensional object, the driver is notified of the presence of the three-dimensional object with a predetermined voice. Therefore, even if the driver is not looking at the monitor 6, the presence of a three-dimensional object can be presented to the driver. Further, even when the three-dimensional object of the first overhead image data is not displayed in the displayed second overhead image data, the three-dimensional object of the first overhead image data is notified to the driver in an easily understandable manner. Can do.

  The above-described embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and various modifications can be made depending on the design and the like as long as the technical idea according to the present invention is not deviated from this embodiment. Of course, it is possible to change.

It is a block diagram which shows the structure of the vehicle periphery image provision apparatus to which this invention is applied. It is a figure which shows an example of the bird's-eye view image obtained by the coordinate transformation of the vehicle periphery image provision apparatus to which this invention is applied. It is a figure explaining the bird's-eye view image data when the solid object on the ground and the three-dimensional object in the air are image | photographed by the vehicle periphery image provision apparatus to which this invention is applied. It is a figure which shows the 2nd bird's-eye view image data produced by the vehicle periphery image provision apparatus to which this invention is applied. It is a figure which shows the image after performing an edge detection process with respect to the 2nd bird's-eye view image data produced by the vehicle periphery image provision apparatus to which this invention is applied. It is a figure explaining the bird's-eye view image data when the solid object on the ground is image | photographed in the vehicle periphery image provision apparatus to which this invention is applied. In the vehicle periphery image provision device to which the present invention is applied, it is another diagram for explaining overhead image data when the host vehicle approaches a three-dimensional object on the ground. It is a figure explaining the process when the own vehicle moves with respect to the solid object on the ground in the vehicle periphery image provision apparatus to which this invention is applied. It is a figure which shows the image data image | photographed by the vehicle periphery image provision apparatus to which this invention is applied. It is a figure explaining the bird's-eye view image data when the air surrounding three-dimensional object was image | photographed in the vehicle periphery image provision apparatus to which this invention is applied. It is a figure explaining the bird's-eye view image data when approaching a three-dimensional object in the air in the vehicle periphery image providing device to which the present invention is applied. It is a figure explaining the process when the own vehicle moves with respect to the three-dimensional object in the air in the vehicle periphery image provision apparatus to which this invention is applied. In the vehicle periphery image providing apparatus to which the present invention is applied, (a) is an explanatory diagram showing that a three-dimensional object below the imaging position of the camera module can be detected, and (b) is a stereoscopic image below and above the imaging position of the camera module. It is explanatory drawing that an object can be detected. 7 is a flowchart showing a processing procedure for detecting a three-dimensional object existing around the host vehicle in the vehicle periphery image providing device to which the present invention is applied.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2a, 2b, 2c, 2d Camera module 3 Vehicle speed sensor 4 Steering angle sensor 5 Shift position sensor 6 Monitor 7 Speaker 10 Communication line 11a, 11b, 11c, 11d Input buffer 12 CPU
DESCRIPTION OF SYMBOLS 13 Image processing part 13a 1st image conversion part 13b 2nd image conversion part 13c Feature point setting part 13d Movement amount detection part 13e Solid determination part 14 Table memory | storage part 14a 1st address conversion table 14b 2nd address conversion table 15 Output buffer 101 Overhead image 102a-102d Vehicle peripheral image 103 White line 104a-104d Partial image 105 Own vehicle mark 301,701,801 Own vehicle 302,702,802 Imaging position 303,703,803 Three-dimensional obstacle 304,704 Horizontal reference surface 401 Overhead image 402 Self-vehicle image 403 White line image 404 Pole image 405 Other vehicle image 501 Edge image 502 Own vehicle image 503 White line edge 504 Pole edge 505 Other vehicle edge 601 Image data 602 Ground partial image 603 Aerial portion Image

Claims (11)

Photographing means for photographing the periphery of the vehicle;
A horizontal plane that is higher than the height of the photographing position of the photographing means is used as a reference plane, and an image in a range higher than the imaging position of the photographing means is subjected to coordinate transformation to generate first overhead image data projected on the reference plane. 1 image conversion means;
First feature point setting means for setting a first feature point in the first overhead image data created by the first image conversion means;
A movement amount detecting means for detecting a movement amount of the host vehicle;
When the host vehicle moves, the movement amount of the host vehicle detected by the movement amount detection unit and the change in the first overhead image data of the first feature point set by the first feature point setting unit; From the above, a solid determination unit that determines whether or not the first feature point is set to a solid object that exists at a position higher than the height of the shooting position of the shooting unit;
An informing means for informing when the solid judging means judges that the first feature point is set to a solid object existing at a position higher than the height of the photographing position of the photographing means; A vehicle periphery image processing apparatus comprising:   The three-dimensional determination means is based on the movement amount of the host vehicle detected by the movement amount detection means and the movement amount in the first overhead image data of the first feature point set by the first feature point setting means. 2. The vehicle periphery image according to claim 1, wherein it is determined whether or not the first feature point is set to a three-dimensional object existing at a position higher than a height of a photographing position of the photographing unit. Processing equipment.   When the solid determination unit determines that the first feature point is set to a three-dimensional object that exists at a position higher than the height of the shooting position of the shooting unit, the high level of the shooting position of the shooting unit is determined. And the movement amount of the host vehicle detected by the movement amount detection means and the movement amount in the first overhead image data of the first feature point set by the first feature point setting means. The vehicle surrounding image processing device according to claim 1, wherein the height of one feature point is obtained.   The solid determination unit detects the movement amount detection unit when determining that the first feature point is set to a three-dimensional object existing at a position higher than the height of the shooting position of the shooting unit. The movement amount of the own vehicle, the movement amount of the first feature point set by the first feature point setting means in the first overhead image data, and the first feature point set by the first feature point setting means The actual distance between the vehicle and the three-dimensional object is obtained from the distance in the first overhead image data between one feature point and the imaging position of the photographing means. The vehicle periphery image processing apparatus described.   The first image conversion means creates the first overhead image data by setting the height of the reference plane to be substantially the same as the height of the vehicle. The vehicle periphery image processing device according to claim 4. Further comprising second image conversion means for creating second overhead image data obtained by coordinate conversion of an image captured by the imaging means with the ground as a reference plane;
The feature point setting means sets a second feature point in the second overhead image data created by the second image conversion means,
When the host vehicle moves, the three-dimensional determination unit is configured to calculate the second feature point based on a movement amount of the host vehicle detected by the movement amount detection unit and a change in the second overhead image data of the second feature point. 6. The vehicle peripheral image processing device according to claim 1, wherein the feature point is determined as a solid object on the ground.   7. The first image conversion unit creates the first overhead image data by setting the height of the reference plane to twice the imaging position of the imaging unit. The vehicle periphery image processing device described in 1.   The feature point setting means refers to color information or brightness information of each pixel on the first overhead image data or the second overhead image data, and the first overhead image data or the second overhead image. An edge line segment in image data is detected, and the first feature point or the second feature point is set in a region on the detected edge line segment or surrounded by the detected edge line segment. The vehicle periphery image processing device according to any one of claims 1 to 7. And further comprising display means for displaying second overhead image data obtained by coordinate conversion by the second image conversion means,
The display means highlights the three-dimensional object when the three-dimensional determination means determines that the first feature point or the second feature point is set as a three-dimensional object. The vehicle periphery image processing device according to any one of claims 6 to 8.   The vehicle surrounding image processing apparatus according to any one of claims 1 to 8, wherein the notification unit further includes a voice output unit that notifies the driver with a predetermined voice. A vehicle periphery image obtained by photographing the periphery of the vehicle is generated by the imaging means, and a horizontal plane higher than the height of the imaging position of the imaging means is used as a reference plane, and an image in a range higher than the imaging position of the imaging means is subjected to coordinate conversion and Create overhead image data projected on the reference plane, set feature points in the created overhead image data,
When the own vehicle moves, the feature point is present at a position higher than the height of the photographing position of the photographing means, based on the amount of movement of the own vehicle and the change in the overhead image data of the set point. A vehicle surroundings state presentation method characterized by determining whether or not a three-dimensional object is present by determining whether or not a three-dimensional object is set.

Images