JP4847303B2 - Obstacle detection method, obstacle detection program, and obstacle detection apparatus - Google Patents

Description

  The present invention relates to an obstacle detection method, an obstacle detection program, and an obstacle detection device for detecting an obstacle that affects the running of a vehicle from an image photographed by a camera, particularly when a single camera is used. The present invention relates to an obstacle detection method, an obstacle detection program, and an obstacle detection apparatus that can accurately detect an obstacle in response to a change in environment.

  In recent years, since the number of injuries caused by traffic accidents has been increasing, various techniques have been devised and put into practical use in order to prevent such traffic accidents. Among such technologies, by detecting obstacles (pedestrians, traveling vehicles, etc.) entering the runway from the video of the on-board camera and notifying the driver, the attention to the obstacles is urged, There is a technology to prevent traffic accidents.

  As a technique for detecting an obstacle, a stereo camera technique for detecting an obstacle using spatial parallax of images taken by a plurality of cameras has been disclosed (for example, see Patent Document 1). However, in the case of detecting an obstacle using the stereo camera technology, it is necessary to install a plurality of cameras on the vehicle, which causes problems such as a reduction in the degree of freedom of vehicle body design and an increase in cost.

  As a technique for solving the above-described problem, there is a technique of detecting an obstacle by using a single camera without using a plurality of cameras (for example, Patent Document 2 and Patent Document 3). In this technique, a motion vector between images at different times is calculated using a single camera, and an obstacle is detected from a difference in motion vectors between images at each time.

  Specifically, the technique disclosed in Patent Document 2 creates a transformation matrix in which feature points in the road surface area extracted from each image overlap and superimpose the images, and image areas that do not overlap well as obstacles Detected.

  The technique disclosed in Patent Document 3 calculates a motion vector of a pixel by dividing an image at different times into regions, and calculates the amount of change in similarity of the image. In addition, an area with a small amount of motion vector and a large amount of change in similarity is detected as an obstacle.

JP 2001-41741 A JP 2003-44996 A Japanese Patent Laid-Open No. 2005-100204

  However, in the above-described conventional technique, when an image is taken with a single camera and an obstacle is detected, the road texture changes due to the vehicle traveling, or the weather changes (reflection, shadow, puddle), etc. There is a problem that obstacles cannot be detected with high accuracy due to large environmental changes.

  The present invention has been made to solve the above-described problems caused by the prior art, and based on an image from a single camera, an obstacle capable of detecting an obstacle with high accuracy even with respect to environmental changes. An object is to provide an object detection method, an obstacle detection program, and an obstacle detection apparatus.

  In order to solve the above-described problems and achieve the object, the present invention is an obstacle detection method for detecting an obstacle that affects the running of a vehicle from an image photographed by a camera, at a different time from the camera. An image storage step of acquiring and storing the image in a storage device, a motion vector calculation step of calculating a motion vector in each region of the image based on images at different times stored in the storage device, and the vehicle A virtual motion vector calculation step of generating a virtual motion vector corresponding to each region of the image based on the generated model based on the generated model; An obstacle detection step of detecting an obstacle based on a virtual motion vector.

  In the invention described above, the virtual motion vector calculation step corrects the virtual motion vector based on a rotation angle of the vehicle.

  Further, in the present invention according to the above invention, the virtual motion vector calculation step converts coordinates on the image into coordinates on the model based on an image pitch of the camera, and generates a virtual corresponding to each region of the image. A characteristic motion vector is calculated.

  Further, according to the present invention, in the above invention, the obstacle detection step detects a difference between the motion vector for each area of the image and the virtual motion vector corresponding to each area, and based on the difference. And detecting obstacles.

  Further, the present invention is an obstacle detection program for detecting an obstacle that affects the running of a vehicle from an image taken by a camera, and acquires an image at a different time point from the camera and stores it in a storage device. An image storage procedure, a motion vector calculation procedure for calculating a motion vector in each region of the image based on images stored at different times stored in the storage device, and a space in which the vehicle travels from the travel state of the vehicle. A virtual motion vector calculation procedure for generating a virtual motion vector corresponding to each region of the image based on the generated model, and an obstacle based on the motion vector and the virtual motion vector An obstacle detection procedure to be detected is executed by a computer.

  Further, the present invention is an obstacle detection device that detects an obstacle that affects the running of a vehicle from images taken by a camera, acquires images at different times from the camera, and at different times obtained. A motion vector calculating means for calculating a motion vector in each region of the image based on the image of the image, and generating a model of a space in which the vehicle travels from the traveling state of the vehicle, and each of the images based on the generated model A virtual motion vector calculating means for calculating a virtual motion vector corresponding to the area; and an obstacle detecting means for detecting an obstacle based on the motion vector and the virtual motion vector. To do.

  According to the present invention, images at different points in time are acquired from the camera, stored in a storage device, motion vectors in each region are calculated based on the images at different points in time stored in the storage device, and the running state of the vehicle A model of the space in which the vehicle travels is generated from the image, virtual motion vectors corresponding to each region of the image are calculated based on the generated model, and obstacles are detected based on the motion vectors and virtual motion vectors Therefore, even when a single camera is used, it is possible to detect obstacles with high accuracy in response to environmental changes.

  Further, according to the present invention, the virtual motion vector is corrected based on the rotation angle of the vehicle, so that the obstacle can be detected more accurately.

  Further, according to the present invention, the coordinates on the image are converted into the coordinates on the model based on the image pitch of the camera, and the virtual motion vector corresponding to each area of the image is calculated. Therefore, the accuracy of the obstacle detection process is improved.

  According to the present invention, the difference between the motion vector for each region of the image and the virtual motion vector corresponding to each region is detected, and the obstacle is detected based on the difference. Obstacles can be detected.

  Exemplary embodiments of an obstacle detection method, an obstacle detection program, and an obstacle detection device according to the present invention will be described below in detail with reference to the accompanying drawings.

  First, the outline and features of the obstacle detection apparatus according to the first embodiment will be described. FIG. 1 is an explanatory diagram for explaining the outline and features of the obstacle detection apparatus according to the first embodiment. As illustrated in FIG. 1, the obstacle detection apparatus according to the first embodiment acquires images at different points in time from the camera, and based on the acquired images, motion vectors (upper row in FIG. 1) for each region of the image. (See the right side) and a virtual motion vector (see the upper left side in FIG. 1) from the running state of the vehicle. Then, the obstacle detection device associates the motion vector with the virtual motion vector and compares them for each region, and from the difference between the motion vector and the virtual motion vector, the obstacle (pedestrian, traveling vehicle, runway) An obstacle entering the inside is detected (see the lower part of FIG. 1).

  As described above, the obstacle detection apparatus according to the first embodiment detects an obstacle based on the motion vector calculated from the images at different time points and the virtual motion vector calculated from the running state of the vehicle. Even when an image from a single camera is used, an obstacle can be detected accurately with respect to environmental changes.

  Next, the configuration of the obstacle detection apparatus according to the first embodiment will be described. FIG. 2 is a functional block diagram of the configuration of the obstacle detection apparatus 100 according to the first embodiment. As shown in the figure, the obstacle detection apparatus 100 includes an image input unit 110, an image storage unit 120, a motion vector calculation unit 130, a camera data input unit 140, a vehicle data input unit 150, a traveling space. A model generation unit 160, a model difference vector calculation unit 170, an obstacle detection unit 180, and an output unit 190 are provided.

  The image input unit 110 is a processing unit that acquires image data (hereinafter referred to as image data) from an in-vehicle camera (not shown) at a constant frame rate and sequentially stores the acquired image data in the image storage unit 120. The image storage unit 120 is a storage unit that stores image data. The image storage unit 120 includes a buffer for storing as much image data as necessary to create a motion vector based on a time difference in an image.

  The motion vector calculation unit 130 acquires image data at different points in time from the image storage unit 120 (for example, image data stored last in the image storage unit 120 and an image taken a predetermined time before this image data) And a motion vector in each area of the image data is calculated based on the acquired image data. As a method for calculating the motion vector, any conventional method may be used. For example, the motion vector can be calculated using a method disclosed in Patent Document 3. The motion vector calculation unit 130 outputs the calculated motion vector data to the model difference vector calculation unit 170.

  The camera data input unit 140 is a processing unit that acquires various data related to a camera installed in the vehicle. Various data related to the camera includes the camera vertical installation position indicating the height from the camera to the road surface, the travel area width indicating the lateral width of the space when the vehicle travels, and the space (vehicle travels from the camera vertical installation position). Data on the traveling area height indicating the height to the upper surface of the (space), the camera focal length indicating the distance from the camera to the focal position of the camera, and the camera pixel pitch indicating the pixel pitch of the camera. Various parameters relating to the camera may be changed dynamically, but the user may set various parameters in advance. The camera data input unit 140 outputs various data related to the camera to the traveling space model generation unit 160.

  The vehicle data input unit 150 is a processing unit that acquires various data related to the running state of the vehicle. Various types of data relating to the running state of the vehicle include vehicle speed data indicating the speed of the vehicle, and vehicle rotation angle data indicating the yaw, roll, and pitch of the vehicle by steering and a slope. The vehicle speed data can be obtained from a vehicle speed pulse and the vehicle rotation angle can be obtained from a steering sensor or a gyroscope. The vehicle data input unit 150 outputs various data related to the traveling state of the vehicle to the traveling space model generation unit 160.

  The traveling space model generation unit 160 is a processing unit that calculates a virtual motion vector (hereinafter referred to as a virtual motion vector) based on various data regarding the camera and various data regarding the traveling state of the vehicle. Hereinafter, processing performed by the traveling space model generation unit 160 will be described in order.

  First, the traveling space model generation unit 160 generates a traveling space model based on various data regarding the camera and various data regarding the traveling state of the vehicle. The traveling space model is a virtual traveling environment of a vehicle, and is a model in which a space where the vehicle will travel locally in the future is included so as to include the imaging range of the camera. It will be predicted that the vehicle will travel from now on is predicted based on the steering angle (vehicle rotation angle) and vehicle speed (vehicle speed data) of the vehicle, and is essentially independent of the white line on the road and the lane width. Therefore, it is possible to generate a traveling space model that is resistant to environmental changes.

  FIG. 3 is a diagram illustrating an example of a traveling space model. In the example shown in FIG. 3, a traveling space model at the time of straight ahead when an in-vehicle camera is arranged at the front center of the vehicle and the photographing range is rectangular is shown. In FIG. 3, h represents the camera vertical installation position, L represents the travel area width, and H represents the travel area height. In the first embodiment, the camera vertical installation position h is described as a negative number. This is because the position where the camera is installed is the origin, and the upward direction from the origin is positive. Further, as described above, the travel area width L and the travel area height H are not defined by the vehicle width and the vehicle height itself, but are defined with some margins.

  The right side of FIG. 3 shows a traveling space model assumed to be viewed from the vehicle-mounted camera according to the first embodiment. As shown in the figure, the space model viewed from the in-vehicle camera can be classified into an upper surface portion, a lower surface portion, and a side surface portion.

  Subsequently, the travel space model generation unit 160 calculates all virtual motion vectors corresponding to the motion vectors calculated by the motion vector calculation unit 130 using the travel space model. Here, for convenience of explanation, a method for calculating a virtual motion vector will be described with the coordinates on the image corresponding to the motion vector as (x, y). In addition, since the calculation method of the virtual motion vector related to the coordinates is only changed, the description thereof is omitted.

  The traveling space model generation unit 160 converts the coordinates (x, y) in the image coordinate system into coordinates (u, v) in the model space coordinate system with the camera installation position as the origin. FIG. 4 is an explanatory diagram for explaining a process of converting coordinates (x, y) in the image coordinate system into coordinates (u, v) in the model space coordinate system. Note that FIG. 4 shows coordinate conversion in the case where the vanishing point is at the center of the image.

によって表すことができる。なお、式（１）および式（２）に含まれるｃは、カメラの画素ピッチを示す。 Width shown in the image coordinate system indicates the number of pixels included in the horizontal width of the image (hereinafter referred to as the number of horizontal pixels of the image), and Height indicates the number of pixels included in the vertical width of the image (hereinafter referred to as the number of vertical pixels of the image). Is shown. A specific formula for converting the coordinates (x, y) of the image coordinate system into the coordinates (u, v) of the model space coordinate system is as follows:

Can be represented by In addition, c contained in Formula (1) and Formula (2) shows the pixel pitch of a camera.

  Subsequently, the traveling space model generation unit 160 determines which region (upper surface portion, lower surface portion, side surface portion; see FIG. 3) of the traveling space model includes the coordinates (u, v) of the model space coordinate system. (Area determination is performed). FIG. 5 is an explanatory diagram for explaining region determination performed by the traveling space model generation unit 160.

を満たす場合に、座標（ｕ，ｖ）が下面部に位置すると判定する。 The traveling space model generation unit 160 determines the position of the coordinates (u, v) based on the relationship between the coordinates (u, v) in the model space coordinate system, the camera vertical installation position h, the traveling region width L, and the traveling region height H. The area to be determined is determined. Specifically, the travel space model generation unit 160 has a relationship of coordinates (u, v), camera vertical installation position h, travel region width L, and travel region height H.

When satisfy | filling, it determines with a coordinate (u, v) being located in a lower surface part.

を満たす場合に、座標（ｕ，ｖ）が上面部に位置すると判定する。なお、走行空間モデル生成部１６０は、座標（ｕ，ｖ）、カメラ垂直設置位置ｈ、走行領域幅Ｌ、走行領域高さＨが式（１）および式（２）のいずれの式にも該当しない場合には、座標（ｕ，ｖ）が側面部に位置すると判定する。 On the other hand, the travel space model generation unit 160 has a relationship of coordinates (u, v), camera vertical installation position h, travel area width L, and travel area height H.

If the condition is satisfied, it is determined that the coordinates (u, v) are located on the upper surface. In the traveling space model generation unit 160, the coordinates (u, v), the camera vertical installation position h, the traveling region width L, and the traveling region height H correspond to both the expressions (1) and (2). If not, it is determined that the coordinates (u, v) are located on the side surface.

  The traveling space model generation unit 160 calculates a virtual motion vector of coordinates (u, v) located in each region based on the determination result by region determination. Hereinafter, a virtual motion vector when the coordinates (u, v) are located on the lower surface, a virtual motion vector when located on the upper surface, and a calculation method of the virtual motion vector when located on the side will be described in order.

(When coordinates (u, v) are located on the bottom surface)
A method of calculating the virtual motion vector (du, dv) when the coordinates (u, v) are located on the lower surface will be described. FIG. 6 is a diagram for supplementing the calculation method of the virtual motion vector when the coordinates (u, v) are located on the lower surface. 6 indicates the focal position of the camera, f indicates the focal distance from the camera to the focal point, and z indicates the distance in the traveling direction from the focal point F to the position A of the object (object candidate for obstacle) ( That is, it indicates from the focal position F to the position B.

  Further, V represents the vehicle speed, dz represents the distance of the object moving in a minute time (time corresponding to the sample rate at which the camera captures an image) dt (in FIG. 6, the vehicle moves from left to right, and the object is Moving from right to left). Further, when the position where the object A moves after dt time is defined as A ′, and the position where the straight line including the position A ′ and the focal position F intersects the vertical axis (v axis) is defined as C, v and The difference from the position C is dv.

となる関係式（５）を導くことができ、式（５）より、

を導くことができる。 As shown in FIG. 6, from the similarity of triangles,

The following relational expression (5) can be derived. From the expression (5),

Can guide you.

を導くことができ、
式（８）から座標（ｕ，ｖ）のｖ軸成分の仮想動きベクトルｄｖを

によって表すことができる。 And by differentiating equation (6) by z,

Can lead
From equation (8), the virtual motion vector dv of the v-axis component of coordinates (u, v) is

Can be represented by

によって表すことができ、式（９）および式（１０）から座標（ｕ，ｖ）のｕ軸成分の仮想動きベクトルｄｕを

によって表すことができる。 On the other hand, the relational expression of u and v in coordinates (u, v) is

The virtual motion vector du of the u-axis component of the coordinates (u, v) from the equations (9) and (10) can be expressed by

Can be represented by

によって表すことができる。 (When coordinates (u, v) are located on the top surface)
Next, a method for calculating the virtual motion vector (du, dv) when the coordinates (u, v) are located on the upper surface will be described. When the coordinates (u, v) are located on the upper surface, it can be obtained by changing the camera vertical installation position h included in the above formulas (5) to (11) to the travel area height H. Specifically, the virtual motion vector when the coordinates (u, v) are located on the upper surface is represented by (du, dv).

Can be represented by

(When coordinates (u, v) are located on the side surface)
Next, a method for calculating the virtual motion vector (du, dv) when the coordinates (u, v) are located on the side surface will be described. FIG. 7 is a diagram for supplementing the method of calculating the virtual motion vector when the coordinates (u, v) are located on the side surface. 7 indicates the focal position of the camera, f indicates the focal distance from the camera to the focal point, and z indicates the distance from the focal point F to the horizontal component E of the object (object that is a candidate for an obstacle) D. Show.

によって表すことができる。 Further, V indicates the vehicle speed, dz indicates the distance of the object moving in a minute time (time corresponding to the sample rate at which the camera captures an image) dt (in FIG. 7, the vehicle moves from left to right, and the object is Moving from right to left). Further, when the position where the object D moves after dt time is defined as D ′, and the position where the straight line including the position D ′ and the focal position F intersects with the u axis is defined as G, the difference between v and the position G is expressed as follows. Assuming dv, the virtual motion vector (du, dv) in the case where the coordinates (u, v) are located on the side surface is as shown in FIG.

Can be represented by

  Returning to the description of FIG. 2, the travel space model generation unit 160 calculates the virtual motion vector corresponding to each coordinate, and then outputs data in which the coordinate and the virtual motion vector are associated to the model difference vector calculation unit 170. . FIG. 8 is a diagram illustrating an example of a virtual motion vector calculated by the traveling space model generation unit 160. The virtual motion vector calculated as a result by the traveling space model generation unit 160 means a motion in the image when a stationary object exists on the boundary of the defined traveling space. Therefore, for example, when the vehicle speed increases, the amount of motion between images increases and the value of the virtual motion vector also increases. Also, as shown in FIG. 8, the value of the virtual motion vector increases as the vehicle approaches.

  The model difference vector calculation unit 170 compares the motion vector acquired from the motion vector calculation unit 130 with the virtual motion vector acquired from the travel space model generation unit 160 for each region, and compares the motion vector for each region. It is a processing unit that calculates a difference vector that is a difference from the virtual motion vector. When a plurality of coordinates are included in the region corresponding to the motion vector and a plurality of virtual motion vectors correspond to a single motion vector, a difference vector is calculated using the average value of the virtual motion vectors. Alternatively, a representative virtual motion vector may be determined to calculate a difference vector. The model difference vector calculation unit 170 outputs the difference vector data for each region to the obstacle detection unit 180.

  The obstacle detection unit 180 is a processing unit that acquires difference vector data and detects an obstacle based on the acquired difference vector data. Specifically, the obstacle detection unit 180 determines a vector that is a detection candidate and a vector that is not a detection candidate based on a difference vector of each region, and detects an obstacle based on a vector distribution of the detection candidate vectors. . As a method for detecting an obstacle from the vector distribution, a conventional detection method may be used.

  Here, a method for determining a vector that is not a detection candidate and a vector that is a detection candidate will be described. When the difference vector between the motion vector and the virtual motion vector is less than the threshold value, the obstacle detection unit 180 determines that the difference vector is a distant view or fine noise, and excludes this. In particular, when the difference vector is 0, the object is a stationary object on the travel space boundary.

  On the other hand, when the difference vector between the motion vector and the virtual motion vector is greater than or equal to the threshold, the obstacle detection unit 180 determines the difference vector as a detection candidate vector. The obstacle detection unit 180 outputs the detected obstacle data to the output unit 190 as obstacle notification data.

  The output unit 190 is a processing unit that outputs various types of information to a monitor (or display, touch panel), speaker, or the like. For example, when the obstacle notification data is acquired from the obstacle detection unit 180, the output unit 190 displays the acquired obstacle notification data on the display. FIG. 9 is a diagram illustrating an example of obstacle notification data displayed on the display. As shown in the figure, the area corresponding to the obstacle is marked.

  Next, a processing procedure of the obstacle detection apparatus 100 according to the first embodiment will be described. FIG. 10 is a flowchart of the process procedure of the obstacle detection apparatus 100 according to the first embodiment. As shown in the figure, the image input unit 110 acquires image data (step S101), stores the image data in the image storage unit 120 (step S102), and the motion vector calculation unit 130 stores the image data at different points in time. A motion vector is obtained from the image storage unit 120 and calculated for each region (step S103).

  The camera data input unit 140 acquires various data related to the camera (step S104), the vehicle data input unit 150 acquires various data related to the running state of the vehicle (step S105), and the traveling space model generation unit 160 uses the camera. A travel space model is generated based on the various data relating to the above and the various data relating to the running state of the vehicle (step S106), and a virtual motion vector of coordinates corresponding to the motion vector region is calculated (step S107).

  Subsequently, the model difference vector calculation unit 170 compares the motion vector acquired from the motion vector calculation unit 130 and the virtual motion vector acquired from the travel space model generation unit 160 for each region, and determines the motion vector for each region. A difference vector that is a difference from the virtual motion vector is calculated (step S108), and the obstacle detection unit 180 detects an obstacle based on the difference vector for each region (step S109), and the obstacle data is obtained. The data is output to the output unit 190 (step S110).

  As described above, the model difference vector calculation unit 170 calculates the difference vector from the motion vector and the virtual motion vector, and the obstacle detection unit 180 detects the obstacle based on the difference vector. Obstacles can be detected.

  As described above, in the obstacle detection apparatus 100 according to the first embodiment, the image input unit 110 sequentially stores image data in the image storage unit 120, and the motion vector calculation unit 130 is stored in the image storage unit 120. A motion vector is calculated based on image data at different times. Then, the travel space model generation unit 160 acquires various data from the camera data input unit 140 and the vehicle data input unit 150 to generate a travel space model, calculates a virtual motion vector from the generated travel space model, and generates a model difference vector. Since the calculation unit 170 calculates a difference vector from the motion vector and the virtual motion vector, and the obstacle detection unit 180 detects an obstacle from the difference vector, even when a single camera is used, the influence of environmental changes Obstacles can be detected with high accuracy without receiving any damage. Further, since it is not necessary to use a plurality of cameras, the cost can be reduced and the degree of freedom in vehicle design can be improved.

  In addition, when the obstacle detection unit 180 detects an obstacle according to the size of the difference vector, mask processing is performed to remove the difference vector whose difference vector value is less than the threshold value, so unnecessary operations are omitted. It is possible to reduce the burden on the obstacle detection process of the obstacle detection unit 180.

  Although the embodiments of the present invention have been described so far, the present invention may be implemented in various different forms other than the first embodiment described above. Therefore, another embodiment included in the present invention will be described below as a second embodiment.

(1) Obstacle detection method When the obstacle detection unit 180 detects an obstacle, the obstacle moves outside the boundary in a certain frame and is not actually on the boundary. Although it may be excluded from the obstacle candidates, this is an instantaneous match, and it does not continuously move to the boundary part and does not continuously detect obstacles. For example, an obstacle can be detected appropriately. As a measure for detecting the obstacle more accurately, the obstacle detection may be performed by reading out image frames at different past times instead of the latest image data stored in the image storage unit 120. Also, the obstacle detection unit 180 can improve the accuracy of obstacle detection by using an object tracking process or the like in combination with the obstacle detected in the previous frame. Further, the obstacle detection unit 180 may use not only the difference vector but also feature quantities such as horizontal and vertical edges when detecting the same obstacle region.

(2) Cooperation with vehicle In the above-described first embodiment, the obstacle detection device 100 displays the information of the obstacle on the monitor when the obstacle is detected. However, the present invention is not limited to this. Instead, various processes can be executed in cooperation with the vehicle. For example, when the obstacle detection device 100 detects an obstacle in front of the vehicle, a deceleration command can be output to the control unit that controls the vehicle speed, and the safety of the vehicle can be ensured. Further, when the driver suddenly brakes when the obstacle detection device 100 detects an obstacle ahead, the obstacle is avoided in cooperation with the control unit that controls the vehicle. Handle control may be performed.

(3) Calculation of virtual motion vector reflecting vehicle rotation angle In the first embodiment, the virtual motion vector calculation method when the vehicle is traveling straight has been described. However, depending on the vehicle rotation angle, the virtual motion vector is calculated. The vector value is corrected. Here, as an example, a method of calculating a virtual motion vector that reflects the yaw vehicle rotation angle will be described. FIG. 11 is an explanatory diagram for supplementarily explaining a method for correcting a virtual motion vector.

  The coordinate (u, v) is one coordinate in the model space coordinate system, but the yaw vehicle rotation angle does not depend on v and the correction value is always 0. That is, it becomes the value of the v component dv of the virtual motion vector obtained in the first embodiment. On the other hand, u forms an angle θ at the intersection of the focal point F, which is the focal length f, and the coordinate axis.

より

ここで

であるため、補正量ｄｕは

となる。補正後のｕ成分の仮想動きベクトルは、実施例１で求めた仮想ベクトルのｕ成分ｄｕに補正量ｄｕを加算したものとなる。なお、他の車両回転角（ロール車両回転角、ピッチ車両回転角）も同様に算出することができる。 At this time, if the yaw vehicle rotation angle is dθ, it can be expressed by each rotation θ _y per unit time and minute time dt, so that the correction amount du can be calculated. Below, an equation for deriving the correction amount du is shown.

Than

here

Therefore, the correction amount du is

It becomes. The corrected u-component virtual motion vector is obtained by adding the correction amount du to the u-component du of the virtual vector obtained in the first embodiment. Other vehicle rotation angles (roll vehicle rotation angle, pitch vehicle rotation angle) can be calculated in the same manner.

  FIG. 12 is a diagram illustrating the difference in the virtual motion vector depending on the yaw vehicle rotation angle. The upper row shows the virtual motion vector when the vehicle is rotating right, and the lower row shows the virtual motion vector when the vehicle is turning left.

(4) System configuration, etc. Among the processes described in the present embodiment, all or part of the processes described as being automatically performed can be performed manually or manually. All or a part of the processing described in the above can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-described document and drawings can be arbitrarily changed unless otherwise specified.

  Each component of the illustrated obstacle detection apparatus 100 is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. Furthermore, each processing function performed by each device may be realized by a CPU and a program that is analyzed and executed by the CPU, or may be realized as hardware by wired logic.

  FIG. 13 is a diagram illustrating a hardware configuration of a computer constituting the obstacle detection device 100 illustrated in FIG. The computer includes an input device 30 that receives data input from a user, a monitor 31, a RAM (Random Access Memory) 32, a ROM (Read Only Memory) 33, and a medium reading device 34 that reads a program from a recording medium on which various programs are recorded. , A camera 35 that captures an image, a driving state detection device (corresponding to a steering sensor, a vehicle speed pulse, a gyroscope, etc.) 36 that detects the driving state of the vehicle, a CPU (Central Processing Unit) 37, and an HDD (Hard Disk Drive) 38 are connected by a bus 39.

  The HDD 38 stores an obstacle detection program 38b that exhibits the same function as that of the obstacle detection device 100 described above. The CPU 37 reads out the obstacle detection program 38b from the HDD 38 and executes it, thereby starting the obstacle detection process 37a that realizes the function of the functional unit of the obstacle detection device 100 described above. The obstacle detection process 37a includes an image input unit 110, a motion vector calculation unit 130, a camera data input unit 140, a vehicle data input unit 150, a traveling space model generation unit 160, a model difference vector calculation unit 170, This corresponds to the obstacle detection unit 180 and the output unit 190.

  Also, the HDD 38 stores various data 38a used for each functional unit described in the obstacle detection apparatus 100 described above. The CPU 37 stores various data 38 a in the HDD 38, reads the various data 38 a from the HDD 38 and stores it in the RAM 32, and executes obstacle detection processing using the various data 32 a stored in the RAM 32.

  By the way, the obstacle detection program 38b is not necessarily stored in the HDD 38 from the beginning. For example, a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card inserted into a computer, or a hard disk drive (HDD) provided inside or outside the computer. The obstacle detection program 38b is stored in the "fixed physical medium" of the computer, and also in the "other computer (or server)" connected to the computer via the public line, the Internet, LAN, WAN, or the like. The computer may read out and execute the obstacle detection program 38b from these.

(Appendix 1) An obstacle detection method for detecting an obstacle that affects the running of a vehicle from an image photographed by a camera,
An image storage step of acquiring images at different times from the camera and storing them in a storage device;
A motion vector calculation step of calculating a motion vector in each region of the image based on images at different times stored in the storage device;
A virtual motion vector calculating step of generating a model of a space in which the vehicle travels from the traveling state of the vehicle and calculating a virtual motion vector corresponding to each region of the image based on the generated model;
An obstacle detection step of detecting an obstacle based on the motion vector and the virtual motion vector;
The obstacle detection method characterized by including.

(Supplementary note 2) The obstacle detection method according to supplementary note 1, wherein the virtual motion vector calculation step corrects the virtual motion vector based on a rotation angle of the vehicle.

(Additional remark 3) The said virtual motion vector calculation process converts the coordinate on the said image into the coordinate on the said model based on the image pitch of the said camera, and calculates the virtual motion vector corresponding to each area | region of the said image The obstacle detection method according to appendix 1 or 2, characterized in that:

(Supplementary Note 4) The obstacle detection step detects a difference between the motion vector for each region of the image and the virtual motion vector corresponding to each region, and detects an obstacle based on the difference. The obstacle detection method according to appendix 1, 2, or 3, wherein

(Additional remark 5) It is an obstacle detection program which detects the obstacle which affects driving | running | working of a vehicle from the image image | photographed with the camera,
Image storage procedure for acquiring images at different times from the camera and storing them in a storage device;
A motion vector calculation procedure for calculating a motion vector in each region of the image based on images at different times stored in the storage device;
A virtual motion vector calculation procedure for generating a model of a space in which the vehicle travels from the traveling state of the vehicle, and calculating a virtual motion vector corresponding to each region of the image based on the generated model;
An obstacle detection procedure for detecting an obstacle based on the motion vector and the virtual motion vector;
Is executed by a computer.

(Supplementary note 6) The obstacle detection program according to supplementary note 5, wherein the virtual motion vector calculation procedure corrects the virtual motion vector based on a rotation angle of the vehicle.

(Additional remark 7) The said virtual motion vector calculation procedure converts the coordinate on the said image into the coordinate on the said model based on the image pitch of the said camera, and calculates the virtual motion vector corresponding to each area | region of the said image The obstacle detection program according to appendix 5 or 6, characterized by:

(Supplementary note 8) The obstacle detection procedure detects a difference between the motion vector for each area of the image and the virtual motion vector corresponding to each area, and detects an obstacle based on the difference. The obstacle detection program according to appendix 5, 6 or 7, wherein

(Additional remark 9) It is an obstacle detection apparatus which detects the obstacle which influences driving | running | working of a vehicle from the image image | photographed with the camera,
Motion vector calculation means for acquiring images at different points in time from the camera, and calculating a motion vector in each region of the image based on the acquired images at different points in time;
A virtual motion vector calculating means for generating a model of a space in which the vehicle travels from the traveling state of the vehicle and calculating a virtual motion vector corresponding to each region of the image based on the generated model;
Obstacle detection means for detecting an obstacle based on the motion vector and the virtual motion vector;
An obstacle detection device comprising:

(Supplementary note 10) The obstacle detection apparatus according to supplementary note 9, wherein the virtual motion vector calculation unit corrects the virtual motion vector based on a rotation angle of the vehicle.

(Additional remark 11) The said virtual motion vector calculation means converts the coordinate on the said image into the coordinate on the said model based on the image pitch of the said camera, and calculates the virtual motion vector corresponding to each area | region of the said image The obstacle detection device according to appendix 9 or 10, wherein:

(Additional remark 12) The said obstacle detection means detects the difference between the said motion vector with respect to each area | region of the said image, and the said virtual motion vector corresponding to each said area, and detects an obstruction based on the said difference The obstacle detection device according to appendix 9, 10 or 11, characterized in that.

  As described above, the obstacle detection method, the obstacle detection program, and the obstacle detection apparatus according to the present invention are useful for an obstacle detection system that detects an obstacle that affects the running of a vehicle, and in particular, a camera. This is suitable when the number of vehicles is limited, the degree of freedom in vehicle design is improved, and obstacles need to be detected accurately in response to environmental changes.

BRIEF DESCRIPTION OF THE DRAWINGS It is explanatory drawing for demonstrating the outline | summary and the characteristic of the obstruction detection apparatus concerning the present Example 1. FIG. 1 is a functional block diagram illustrating a configuration of an obstacle detection apparatus according to a first embodiment. It is a figure which shows an example of a driving | running | working space model. It is explanatory drawing for demonstrating the process which converts the coordinate (x, y) of an image coordinate system into the coordinate (u, v) of a model space coordinate system. It is explanatory drawing for demonstrating the area | region determination which a driving | running | working space model production | generation part performs. It is a figure for supplementing the calculation method of a virtual motion vector in case a coordinate (u, v) is located in a lower surface part. It is a figure for supplementing the calculation method of a virtual motion vector in case a coordinate (u, v) is located in a side part. It is a figure which shows an example of the virtual motion vector calculated by the traveling space model production | generation part. It is a figure which shows an example of the obstacle notification data displayed on a display. 3 is a flowchart illustrating a processing procedure of the obstacle detection apparatus according to the first embodiment. It is explanatory drawing for supplementarily demonstrating the correction method of a virtual motion vector. It is a figure which shows the difference in the virtual motion vector by a yaw vehicle rotation angle. It is a figure which shows the hardware constitutions of the computer which comprises the obstruction detection apparatus shown in FIG.

Explanation of symbols

30 Input device 31 Monitor 32 RAM
32a, 38a Various data 33 ROM
34 Medium Reading Device 35 Camera 36 Running State Detection Device 37 CPU
37a Obstacle detection process 38 HDD
38b Obstacle detection program 39 Bus 100 Obstacle detection device 110 Image input unit 120 Image storage unit 130 Motion vector calculation unit 140 Camera data input unit 150 Vehicle data input unit 160 Traveling space model generation unit 170 Model difference vector calculation unit 180 Obstacle Detection unit 190 Output unit

Claims (4)

An obstacle detection method for detecting an obstacle that affects driving of a vehicle from an image captured by a camera,
An image storage step of acquiring images at different times from the camera and storing them in a storage device;
A motion vector calculation step of calculating a motion vector in each region of the image based on images at different times stored in the storage device;
The vertical direction of the camera , which is the height of the traveling area, which is the height in which the vehicle travels in the model, the traveling area width, in which the vehicle travels in the model , and the height where the camera is installed in the model Based on the position, a model of the space in which the vehicle travels is generated. For each coordinate in the image coordinate system, a value corresponding to the number of pixels included in the horizontal or vertical width of the image and the horizontal direction of the coordinate in the image coordinate system Or, by multiplying the result of the predetermined addition and subtraction with the vertical component by the pixel pitch of the camera, the coordinates of the spatial coordinate system are calculated, and the position of each coordinate of the calculated spatial coordinate system is the lower surface portion of the model, By determining which area of the upper surface portion or the side surface portion is located, it is determined whether each coordinate of the spatial coordinate system is positioned on the lower surface portion, the upper surface portion, or the side surface portion of the model, and the upper surface of the model Part For each coordinate of the spatial coordinate system determined to be located in either the lower surface portion or the side surface region, the upper surface portion, the lower surface portion, and the side surface region determined to be positioned correspond to the coordinates of the spatial coordinate system. The value obtained by dividing the value by the focal length is multiplied by the moving distance of the obstacle moving at the time corresponding to the sample rate at which the camera takes an image and corresponding to the change in the distance from the focal distance to the obstacle. The value obtained by dividing the obtained value by the camera vertical installation position, the travel area height, or the travel area width is used as the vertical or horizontal component of the virtual motion vector, so that each of the spatial coordinate systems of the model A virtual motion vector calculation step of calculating a virtual motion vector of the region;
An obstacle detection step of detecting an area where a difference vector between the motion vector and the virtual motion vector is greater than or equal to a threshold as an obstacle area;
The obstacle detection method characterized by including. The virtual motion vector calculation step includes a value obtained by dividing the sum of the square of the focal length and the square of the value of the horizontal component of the spatial coordinate system by the focal length, and the horizontal vehicle per unit time. By calculating the product of the rotation angle and the time corresponding to the sample rate at which the camera captures an image as a correction amount, and adding the correction amount to the horizontal component of the virtual motion vector , the virtual The obstacle detection method according to claim 1, wherein a correct motion vector is corrected. An obstacle detection program for detecting an obstacle that affects driving of a vehicle from an image captured by a camera,
Image storage procedure for acquiring images at different times from the camera and storing them in a storage device;
A motion vector calculation procedure for calculating a motion vector in each region of the image based on images at different times stored in the storage device;
The vertical direction of the camera , which is the height of the traveling area, which is the height in which the vehicle travels in the model, the traveling area width, in which the vehicle travels in the model , and the height where the camera is installed in the model Based on the position, a model of the space in which the vehicle travels is generated. For each coordinate in the image coordinate system, a value corresponding to the number of pixels included in the horizontal or vertical width of the image and the horizontal direction of the coordinate in the image coordinate system Or, by multiplying the result of the predetermined addition and subtraction with the vertical component by the pixel pitch of the camera, the coordinates of the spatial coordinate system are calculated, and the position of each coordinate of the calculated spatial coordinate system is the lower surface portion of the model, By determining which area of the upper surface portion or the side surface portion is located, it is determined whether each coordinate of the spatial coordinate system is positioned on the lower surface portion, the upper surface portion, or the side surface portion of the model, and the upper surface of the model Part For each coordinate of the spatial coordinate system determined to be located in either the lower surface portion or the side surface region, the upper surface portion, the lower surface portion, and the side surface region determined to be positioned correspond to the coordinates of the spatial coordinate system. The value obtained by dividing the value by the focal length is multiplied by the moving distance of the obstacle moving at the time corresponding to the sample rate at which the camera takes an image and corresponding to the change in the distance from the focal distance to the obstacle. The value obtained by dividing the obtained value by the camera vertical installation position, the travel area height, or the travel area width is used as the vertical or horizontal component of the virtual motion vector, so that each of the spatial coordinate systems of the model A virtual motion vector calculation procedure for calculating a virtual motion vector of the region;
An obstacle detection procedure for detecting, as an obstacle area, an area where the magnitude of a difference vector between the motion vector and the virtual motion vector is equal to or greater than a threshold;
Is executed by a computer. An obstacle detection device that detects an obstacle that affects driving of a vehicle from an image captured by a camera,
Motion vector calculation means for acquiring images at different points in time from the camera, and calculating a motion vector in each region of the image based on the acquired images at different points in time;
The vertical direction of the camera , which is the height of the traveling area, which is the height in which the vehicle travels in the model, the traveling area width, in which the vehicle travels in the model , and the height where the camera is installed in the model Based on the position, a model of the space in which the vehicle travels is generated. For each coordinate in the image coordinate system, a value corresponding to the number of pixels included in the horizontal or vertical width of the image and the horizontal direction of the coordinate in the image coordinate system Or, by multiplying the result of the predetermined addition and subtraction with the vertical component by the pixel pitch of the camera, the coordinates of the spatial coordinate system are calculated, and the position of each coordinate of the calculated spatial coordinate system is the lower surface portion of the model, By determining which area of the upper surface portion or the side surface portion is located, it is determined whether each coordinate of the spatial coordinate system is positioned on the lower surface portion, the upper surface portion, or the side surface portion of the model, and the upper surface of the model Part For each coordinate of the spatial coordinate system determined to be located in either the lower surface portion or the side surface region, the upper surface portion, the lower surface portion, and the side surface region determined to be positioned correspond to the coordinates of the spatial coordinate system. The value obtained by dividing the value by the focal length is multiplied by the moving distance of the obstacle moving at the time corresponding to the sample rate at which the camera takes an image and corresponding to the change in the distance from the focal distance to the obstacle. The value obtained by dividing the obtained value by the camera vertical installation position, the travel area height, or the travel area width is used as the vertical or horizontal component of the virtual motion vector, so that each of the spatial coordinate systems of the model Virtual motion vector calculation means for calculating a virtual motion vector of the region;
Obstacle detection means for detecting an area where a difference vector between the motion vector and the virtual motion vector is greater than or equal to a threshold as an obstacle area;
An obstacle detection device comprising:

Images