JP5624370B2 - Moving body detection apparatus and moving body detection method - Google Patents

Description

  The present invention relates to a moving body detection apparatus and a moving body detection method for photographing a video around a vehicle with a wide-angle camera mounted on the vehicle and processing the camera image to detect a moving body.

  Conventionally, a plurality of cameras are mounted on a vehicle, the front, rear or side of the host vehicle is photographed, a moving body is detected based on the photographed image, and a moving body approaching the host vehicle is displayed on the display unit. A vehicular image display device is known. There is also an example in which a wide-field image is taken using a fish-eye camera or the like as a camera constituting the imaging unit.

  Patent Document 1 discloses a method of predicting a motion vector on an image projected on a linear plane from the own vehicle movement parameter, and determining a stationary object or a moving body from a difference from an actually measured vector.

  Patent Document 2 discloses a moving object detection device. In patent document 2, the motion vector of the observation target in the image observed by the moving observation means is extracted, the motion vector is extended to obtain a point where the motion vector concentrates, and the temporal position change of the concentration point is analyzed. A moving object is detected. That is, a linear projection plane is assumed. In the example of Patent Document 2, a stationary object and a moving object can be determined even when the camera position moves like an in-vehicle camera. The object is projected onto a plane, and the projected plane image is displayed on the projected plane image. The motion vector is calculated, the intersection point on the extension line of the motion vector is calculated, and the moving object is determined from the relationship of the intersection point.

  Therefore, both Patent Document 1 and Patent Document 2 have a premise that the projected image is defined on a linear plane, and analysis with some linear prediction is performed based on this premise.

  However, a linear projection plane cannot be defined because a linearly projected plane is infinite with a camera having a viewing angle of 180 degrees, such as a camera using a fisheye camera or a convex mirror as an in-vehicle camera. For this reason, a super-wide-angle camera with a viewing angle exceeding 180 degrees uses a non-linear projection method to project onto a finite plane. Therefore, an analysis method based on a linear projection plane cannot be applied to an ultra-wide-angle camera, and a moving object cannot be detected.

Japanese Patent Laid-Open No. 5-233813 JP-A-8-194822

  In the case of an in-vehicle camera having a viewing angle of 180 degrees or more than 180 degrees, such as a fish-eye camera or a camera using a convex mirror, a nonlinear projection method is used. It cannot be detected.

  In view of such circumstances, an object of the present invention is to provide a moving body detection apparatus and a moving body detection method capable of detecting a moving body even in a non-linear image captured by a camera having a wide-angle imaging range. To do.

  According to an embodiment of the present invention, a moving body detection apparatus divides a nonlinear image captured by a camera mounted on a vehicle and having a wide-angle imaging range into a plurality of images in a preset viewing angle range and has a common camera center. A division conversion unit that generates a plurality of linear projection plane images, a motion vector calculation unit that calculates a motion vector of each of the plurality of projection plane images, and analyzes and moves the motion vectors of the plurality of projection plane images A moving body determination unit that determines a body and a display unit that displays a determination result of the moving body determination unit are provided.

  A moving body detection method according to an embodiment of claim 5, wherein a non-linear image captured by a camera mounted on a vehicle and having a wide-angle imaging range is divided into a plurality of images having a preset viewing angle range, and a common camera center Generating a plurality of linear projection plane images, calculating respective motion vectors of the plurality of projection plane images, analyzing the motion vectors of the plurality of projection plane images, determining a moving body, and moving the movement The body determination result is displayed on the display unit.

  According to the moving body detection apparatus according to the embodiment of the present invention, a moving body is determined based on a motion vector by converting a captured image of a super-wide-angle camera such as a fisheye camera into a plurality of linear divided projection plane images. It becomes possible, the direction of the moving body can be determined, the approach to the own vehicle can be notified, and danger avoidance can be avoided.

The block diagram which shows the structure of the moving body detection apparatus which concerns on one Embodiment of this invention. The block diagram which shows the detailed structure of a moving body detection part. Explanatory drawing which shows roughly the image image | photographed with the fisheye camera. Explanatory drawing explaining operation | movement of a division conversion part. Explanatory drawing which shows a pinhole camera model. Explanatory drawing explaining operation | movement of a moving body determination part. Explanatory drawing explaining operation | movement of the moving body detection apparatus which concerns on 2nd Embodiment.

  Hereinafter, a mobile object detection apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a moving object detection apparatus 100 according to an embodiment of the present invention. In FIG. 1, the moving body detection unit 10 processes an image captured by the camera 20 mounted on the vehicle 1 to detect the moving body, and displays the detection result on the display unit 30.

  The camera 20 is a super-wide-angle camera having a viewing angle exceeding 180, such as a fish-eye lens camera, for example, and is mounted on the vehicle 1 to capture an image around the vehicle. FIG. 1 shows an example in which cameras 20... Are mounted in front and rear of the vehicle 1. In the following description, a case where a fisheye lens camera (referred to as a fisheye camera) is used as the camera 20 will be described.

  FIG. 2 is a block diagram illustrating a detailed configuration of the moving object detection unit 10. The moving body detection unit 10 includes a memory 11, a division conversion unit 12, a motion vector calculation unit 13, a moving body determination unit 14, an original image conversion unit 15, an approach direction determination unit 16, and a control unit 17.

  Hereinafter, functions and operations of each unit of the moving body detection unit 10 will be described. The control unit 17 is a microprocessor including a CPU, a ROM, a RAM, and the like. Each unit of the moving object detection unit 10 (memory 11, division conversion unit 12, motion vector calculation unit 13, moving object determination) according to a program stored in the ROM. Unit 14, the original image conversion unit 15, and the approach direction determination unit 16).

  The camera 20 has an imaging range of 180 degrees or more, images an image projected nonlinearly on the imaging surface, and writes an imaging signal in the memory 11. An image captured by the camera 20 (camera image) is schematically shown as a non-linear image as shown in FIG. 3, for example. FIG. 4 is a plan view showing the imaging range of the camera 20.

  The division conversion unit 12 divides the captured image into three viewing angle ranges as illustrated in FIG. 4 to generate a divided projection plane image. As the camera coordinate system, xyz when the optical axis of the camera 20 is the z-axis, the axis horizontal to the road surface is the x-axis, and the axis perpendicular to the road surface is the y-axis (in FIG. 4, the y-axis is perpendicular to the paper surface). Consider a coordinate system.

  As shown in FIG. 4, three divided projection planes that are in contact with a semicircle having an imaging surface (an image as shown in FIG. 3) projected nonlinearly on the camera on a plane of z = 0 and having a focal length f as a radius. 1, 2, 3 are assumed. These images on the divided projection planes 1, 2, and 3 can be regarded as images taken by three virtual cameras. That is, it has a common camera center O (0, 0, 0), and the optical axes of the three virtual cameras are on the plane where y = 0. In the central virtual camera, the optical axis z coincides with the optical axis of the camera 20, and the optical axes L and R of the left virtual camera and the right virtual camera are ± 60 degrees with respect to the optical axis z of the central virtual camera. Just leaning.

  In the example of FIG. 4, three virtual cameras each cover a viewing angle of 60 degrees + α degrees (α indicates a slight overlap). Therefore, the three virtual cameras are pinhole cameras having a common camera center (O) and different optical axis angles.

  FIG. 5 is an explanatory diagram showing a pinhole camera model. In FIG. 5, reference numeral 200 denotes a projection plane with z = f (f is a focal length), and pixels at a certain point on the projection plane 200 are indicated by P and P ′. The angle of view is 60 degrees + α.

  In each virtual camera, a point on the imaging plane (y = 0) has a one-to-one correspondence with a point on each projection plane within a range of a viewing angle of 60 degrees and α degrees. This correspondence can be obtained from a mathematical expression that determines the projection method of the camera 20. The correspondence relationship obtained in advance may be provided as a conversion table (LUT). The conversion table (LUT) corrects the distortion of the nonlinear captured image, and the captured data is converted into a linear projection plane image based on the conversion table.

  The division conversion unit 12 performs the following conversion process. From the image information of the camera 20 stored in the memory 11, image information corresponding to 60 degrees in the center is selected and read out, and converted into a center plane image using a conversion table (LUT). Similarly, image information corresponding to 60 degrees on the left side is selected and converted into a left plane image using a conversion table (LUT). Also, image information corresponding to 60 degrees on the right side is selected and converted into a right plane image using a conversion table (LUT).

  The motion vector calculation unit 13 calculates the motion vectors of the three planar images.

For the motion vector calculation, a known block matching or gradient method can be used. As is well known, in the pinhole camera model (FIG. 5) with a focal length f, the motion vector (u, v) ^T in the projection plane xy generated on the stationary object subject as the camera (own vehicle) moves is Is expressed by the following equation.

  Here, Rx, Ry, and Rz represent vehicle rotation components around the x axis, the y axis, and the z axis, and Tx, Ty, and Tz represent vehicle translation components in the x, y, and z axis directions.

  x and y represent pixel positions on the image plane, and z represents the distance from the camera to the subject. From this equation, the movement of the image accompanying the movement of the own vehicle (camera 20) is related.

  If the motion vector (u, v) of a certain point P (x, y) on the image is related and the own vehicle movement parameter can be obtained by some method, the motion of the stationary object on the screen can be predicted. If it is as predicted, it can be determined as a stationary object, and if it is not predicted, it can be determined as a moving object. Therefore, by analyzing based on the above equation, it is possible to determine the movement of the stationary object due to the movement of the vehicle and the movement specific to the moving object.

  The mobile body determination unit 14 detects each mobile body on three planes from the motion vectors of the three plane images by linear prediction determination. For example, in the central plane image shown in FIG. 6, when the moving direction of the own vehicle is A, and there is a moving body (for example, a vehicle traveling from the side) showing a moving direction B different from the own vehicle, Judged as a moving object. There are various determination methods for determining a moving object.

  The determination result of the moving body by the moving body determination unit 14 is superimposed on the original camera image by the original image conversion unit 15. An object that moves across a plurality of divided projection planes can also be recognized as a moving body that moves continuously on the original imaging surface.

  The approaching direction determination unit 16 determines the moving direction from the motion vector of the moving body, determines whether or not the vehicle is approaching, and a warning that can be visually recognized by the driver when there is a moving body that is likely to approach. Image information is generated and output to the display unit 30. The warning image is information such as an arrow indicating from which direction the moving body is approaching. Therefore, if there is a moving body such as another vehicle or a person approaching the host vehicle from the front or the rear, the display unit 30 can accurately notify the driver.

  Next, a second embodiment of the present invention will be described. The second embodiment is characterized in the operation of the division conversion unit 12 and divides an image captured by the camera 20 into two viewing angle ranges to generate a divided projection plane image. That is, as shown in FIG. 7, a virtual camera having divided projection planes 1 and 2 in contact with a semicircle having a common camera center O and having a focal length f as a radius is assumed. In FIG. 7, a left virtual camera and a right virtual camera having optical axes L and R of ± 45 degrees with respect to the optical axis z of the camera 20 are assumed.

  By replacing the three-plane division in FIG. 4 with a two-plane division, the same processing is performed thereafter, and the motion vector calculation unit 13 calculates the respective motion vectors of the two plane images. The moving body determination unit 14 detects each moving body on two planes from the motion vectors of the two plane images. The original image conversion unit 15 superimposes the determination result by the moving body determination unit 14 on the original camera image. The approach direction determination unit 16 determines the direction of movement from the motion vector of the moving body, and determines whether or not the vehicle is approaching.

  According to the embodiment described above, it is possible to determine a moving object based on a motion vector by converting a captured image of a wide-angle camera such as a fisheye camera into a plurality of divided projection plane images, and the direction of the moving object Can be notified and approach to the own vehicle can be notified, and danger avoidance can be avoided.

  In the above-described embodiment, the example in which the division conversion unit 12 divides into three or two has been described. Although an example using a fisheye camera as the camera 20 has been described, a camera having a viewing angle of 180 degrees or 180 degrees or more, such as a camera using a convex mirror, can be used. Various modifications can be made without departing from the scope of the claims.

DESCRIPTION OF SYMBOLS 100 ... Moving body detection apparatus 10 ... Moving body detection part 11 ... Memory 12 ... Division | segmentation conversion part 13 ... Motion vector calculation part 14 ... Moving body determination part 15 ... Original image conversion part 16 ... Approaching direction determination part 17 ... Control part 20 ... Camera 30 ... display section

Claims (7)

A division conversion unit that divides a nonlinear image captured by a camera mounted on a vehicle and has a wide-angle imaging range into a plurality of images in a preset viewing angle range and generates a plurality of linear projection plane images having a common camera center; ,
A motion vector calculation unit for calculating a motion vector of each of the plurality of projection plane images;
A moving body determination unit that analyzes the motion vectors of the plurality of projection plane images and determines a moving body;
A display unit for displaying a determination result of the mobile body determination unit;
A moving body detection apparatus comprising:   2. The movement according to claim 1, wherein the division conversion unit generates a plurality of planar projection images by dividing a wide-angle camera image of 180 degrees or more into a plurality of images within 180 degrees having an overlap unit. Body detection device.   2. The moving body detection apparatus according to claim 1, wherein the moving body determination unit determines the presence or absence of a moving body approaching the host vehicle based on the direction of the motion vector calculated by the motion vector calculation unit. . An original image conversion unit that superimposes the determination result of the moving body determination unit on the captured image before the division;
An approach direction determination unit that determines an approach direction of the moving body based on an image of the original image conversion unit and displays image information indicating the approach of the moving body on the display unit. Item 4. A moving body detection apparatus according to Item 1. A nonlinear image captured by a camera mounted on a vehicle and having a wide-angle imaging range is divided into a plurality of images having a preset viewing angle range to generate a plurality of linear projection plane images having a common camera center,
Calculating a motion vector of each of the plurality of projection plane images;
Analyzing the motion vectors of the plurality of projection plane images to determine a moving object;
A moving object detection method, comprising: displaying a determination result of the moving object on a display unit.   6. The moving body detection method according to claim 5, wherein presence or absence of a moving body approaching the host vehicle is determined based on the calculated direction of the motion vector. The determination result of the moving object is superimposed on the original image before division,
Determine the approaching direction of the moving body based on the original image on which the determination result is superimposed,
6. The moving object detection method according to claim 5, wherein image information indicating the approach of the moving object is displayed on the display unit.

Images