JP2007334859A - Object detector - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object detector accurately detecting a stationary object. <P>SOLUTION: A canonical flow generating means 2 generates a canonical flow of an image imagined in a standard environment with no existing object, based on an installation state of an image pick-up means 3 installed in a vehicle and a travel condition of the vehicle, an observation flow generating means 4 generates an observation flow, based on an image picked up by the image pick-up means, and a comparison collating part 5 detects the object, based on a difference between the canonical flow generated by the canonical flow generating means and the observation flow generated by the observation flow generating means. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an object detection apparatus that detects an object based on an image captured by an imaging unit mounted on a moving body such as a vehicle, a mobile robot, or a person.

  In the mobile body, the situation of the movement reference plane is important as a situation to be recognized for ensuring its own safety. The situation of the movement reference plane includes the shape of the plane, the presence / absence of another moving body, and the presence / absence of a stationary object. In addition, road conditions are important for vehicles such as automobiles as situations to be recognized for ensuring their own safety. Examples of the road situation include a road shape such as a curve, the presence or absence of a moving object such as another moving vehicle or a pedestrian, and the presence or absence of a stationary object such as a stopped vehicle.

As an object detection method, a method using distance information and a method using motion information are conceivable. Examples of using distance information include a method using a distance sensor (the following non-patent document 1) and a method using stereo vision (the following non-patent document 2). As a method using motion information, a method using an optical flow and FOE (Focus of Expansion) (the following Non-Patent Document 3) has been studied. Non-patent documents 4 and 5 below are known as conventional examples for obtaining an optical flow that is a movement vector on the screen. Non-Patent Document 5 is called a KLT (Kanade-Lucas-Tomasi) method.
Isao Arata, Tokumi Sase, Kazunobu Umeda, “Proposal of Obstacle Avoidance Method for Mobile Robot Using Small Range Image Sensor”, Proc. Of the Japan Society of Mechanical Engineers Robotics and Mechatronics '01, 1P1-N4, 2001.6. Jun Okada, Misaki Kaga, Masayuki Inaba, Hiroki Inoue, “3D obstacle avoidance for legged robots by integrating color regions and binocular stereo” Proceedings of the 16th Annual Conference of the Robotics Society of Japan, pp 1505-1506, 1998 . Mutsumi Watanabe, Nobuyuki Takeda, Kazunori Onoguchi, “Moving obstacle detection and recognition by optical flow patern analysis for mobilerobot”, Advanced Robotics, Vol.12, No.7,8, pp.791-816, 1999.tcts Jianbo Shi, Carlo Tomasi: “Good Features to Tack”, 1994 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'94), pp.593-600, 1994. ortelrt ” Jean-Yves Bouguet: “Pyramidal Impementation of the Lucas Kanade Feature Tacker Descripion of the algorithm, OpenCV Documentation, Microprocessor Research Labs, Intel Corporation, 1999.

  In the method using the distance sensor, the distance and position from the sensor to the object can be known, and the distance to the object can be obtained with high accuracy. Therefore, practical use such as warning of a collision has been made. However, it is necessary to determine the degree of danger and avoidance behavior according to the target object while traveling, and it is desirable that the detected object can be recognized.

  A method of using stereo vision as a method for obtaining distance information from an image is used for object recognition in a landscape. However, this method can perform processing only at an overlapping portion of two camera images arranged at a certain interval, and the field of view is narrowed. For this reason, a portion near the camera cannot be processed because it cannot have a common field of view. For example, when mounted on a mobile robot, a method that can handle a close object that may collide with a single camera is appropriate. It is believed that there is.

  As a method of using motion information from information as a video, there is a method using optical flow and FOE. It is used to detect moving objects that move differently from the background by examining the difference in FOE. However, since the stationary object has the same FOE as the background, there is a problem that the stationary object cannot be accurately detected.

  An object of the present invention is to provide an object detection device that can accurately detect a stationary object in light of the problems of the conventional example.

In order to achieve the above object, the present invention provides an object detection apparatus for detecting an object based on an image captured by an imaging unit installed on a moving body.
A first optical for generating a first optical flow of a screen assumed in a standard environment where no object exists, based on an installation state of the imaging unit installed on the mobile body and a movement state of the mobile body Flow generation means;
Second optical flow generation means for generating a second optical flow based on an image captured by the imaging means;
Object detection means for detecting an object based on the difference between the generated first and second optical flows;
It is characterized by having.
With this configuration, since the object is detected based on the difference between the first optical flow of the virtual image not including the object and the second optical flow of the captured real image, the stationary object can be accurately detected. .

The first optical flow generation means includes
By setting the height from the reference plane of the imaging means, the focal length of the imaging means, and the inclination of the imaging means with respect to the horizontal plane as parameters, the feature points are arranged on the reference plane in the virtual space,
Acquiring the moving speed and rotational angular velocity of the imaging means at predetermined intervals, changing the coordinates of the feature points, and generating the previous feature point and the current feature point as the start point and end point of the first optical flow, respectively; It is characterized by.
The moving body is a vehicle,
The first optical flow generation means includes:
The height of the imaging means from the road surface, the focal length of the imaging means, the inclination of the imaging means with respect to the horizontal plane, the distance between the front and rear wheels of the vehicle, and the distance between the right and left wheels of the vehicle. By setting the five fixed parameters, feature points are placed on the road surface in the virtual space,
The moving speed of the vehicle and the inclination angle of the front wheel with respect to the longitudinal direction of the vehicle body are received as variable parameters at predetermined time intervals, the coordinates of the feature points are changed, and the previous feature point and the current feature point are respectively changed to the first optical point. It is generated as a start point and an end point of a flow.
With this configuration, it is possible to accurately detect the first optical flow of a virtual image that does not include an object.

In addition, the object detection unit divides the screen imaged by the imaging unit into regions, sets a representative flow of the first optical flow for each region, and includes the generated second optical flow. A flow larger than the representative flow is determined as an object.
Further, the object detection means sets the divided areas by using the screen imaged by the imaging means and information on the start point and end point of the optical flow obtained when the first optical flow is generated. And
With this configuration, it is possible to prevent erroneous detection due to noise.

Further, the object detection means determines that a block in which the number of flows determined to be the object exceeds a threshold is an object block.
With this configuration, it is possible to prevent erroneous detection due to noise.

  According to the present invention, the object is detected based on the difference between the first optical flow of the virtual image that does not include the object and the second optical flow of the captured real image, so that the stationary object is accurately detected. Can do.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of an object detection apparatus according to the present invention, and FIG. 2 is a flowchart for explaining processing of the object detection apparatus of FIG.

  In the present invention, in order to detect a stationary object in a moving state, a concept of “normative flow” is proposed as an alternative method to the conventional method. That is, the optical flow at each location in the image predicted from the standard environment and the standard moving state is obtained by calculation, and this is used as the reference flow (first optical flow). Then, an object is detected by detecting a difference from an optical flow (second optical flow, observation flow) of an actual image obtained by moving image processing.

  FIG. 1 shows an object detection device mounted on a vehicle as an example of a moving body to be mounted. In FIG. 1, vehicle information acquisition means 1 that is movement information acquisition means acquires travel information of a vehicle in which an image imaging means 3 is installed, and normative flow generation means (first optical flow generation means) 2 Based on the installation state of the image pickup means 3 and the vehicle travel information from the vehicle information acquisition means 1, a reference flow (first optical flow) of a screen assumed in a standard environment where no object exists is generated. The observation flow generation means (second optical flow generation means) 4 generates an observation flow (second optical flow) based on the image from the image capturing means 3, and the comparison / collation unit (object detection means) 5 is a norm. An object is detected based on the difference between the flow and the observation flow. Here, the block from the vehicle information acquisition means 1 to the comparison collation part 5 comprises one Embodiment of the object detection apparatus which concerns on this invention. The peripheral object information construction unit 6 performs image processing that allows the user to recognize the display area detected as an object by the comparison / collation unit 5, and combines the image with the image from the image capturing unit 3 and outputs it to the display unit 7.

The process will be described in detail with reference to FIG.
Step S1: First, information such as the speed of the automobile and the turning angle of the steering wheel is received as information such as the moving speed and rotational angular speed of the image pickup means 3, and the coordinates of the feature points arranged on the road surface in the virtual space are moved. Then, a normative flow generation process is performed to generate an optical flow in which the movement is projected on the screen. The optical flow generated by this standard flow generation process is defined as a standard flow.
Step S2: Next, the flow (observation flow) of the real image is extracted by the KLT method in Non-Patent Documents 4, 5, etc.
Step S3: Next, the screen is divided into blocks, the length and direction of the reference flow are set for each block, the length and direction of the flow of the actual image are checked, and there is a difference from the reference flow in the block containing the flow. Check for any.
Step S4: If there is a flow in the block that is finally determined to have a difference, it is detected that an object exists in the block.
For each process, (1) generation of the normative flow, (2) feature point extraction and flow detection in the input image, (3) collation between the normative flow and the actual flow, and (4) object detection are specifically described in this order. explain.

(1) Generation of normative flow The normative flow generation process that is the basis of the present invention will be described. First, as fixed parameters for generating the normative flow, the height of the image pickup means 3 from the road surface of the camera, the focal length of the camera, the tilt of the camera with respect to the horizontal, the distance between the front and rear wheels of the vehicle, the right side of the vehicle By setting five parameters of the distance between the wheel and the left wheel, feature points are arranged on the road surface in the virtual space.
Next, as the variable parameters for generating the normative flow, the current moving speed of the vehicle and the inclination angle of the front wheels with respect to the longitudinal direction of the vehicle body are received every predetermined time, and the coordinates of the feature points are changed according to the running state of the vehicle. . The position of the feature point is calculated from the information of the fixed parameter set at the beginning, and where it is projected on the camera image is calculated, and the previous state and the current state are output as the start point and end point of the reference flow, respectively.
If the moving body to be mounted is not a vehicle, the virtual image can be obtained by setting three parameters: the height of the image pickup means 3 from the camera movement reference plane, the focal length of the camera, and the tilt of the camera with respect to the horizontal. Feature points are placed on a reference plane in space. Next, as the variable parameters for generating the reference flow, the current moving speed and rotational angular velocity of the imaging unit are received every predetermined time, and the coordinates of the feature points are changed according to the moving state of the imaging unit. The position of the feature point is calculated from the information of the fixed parameter set at the beginning, and where it is projected on the camera image is calculated, and the previous state and the current state are output as the start point and end point of the reference flow, respectively.

<Calculation of screen projection>
The space on the reference flow generation process and the position on the screen obtained by the image pickup device are approximated to a pinhole camera, the viewpoint is the origin as shown in FIG. 3, the x axis is the imaging plane and the width direction of the screen, and the y axis When the space with depth is considered on the xy plane, if the focal length of the camera is f, the distance from the imaging surface to the screen is y = f (1)
Is a straight line. The range of the screen is the width of the imaging surface with the center of the imaging surface as the y axis. The point projected on the screen is the intersection of this straight line and the straight line connecting the coordinates of the feature point and the origin.

Further, in FIG. 3, assuming that the coordinates of the feature points that are actual points are (Ox, Oy) and the coordinates of the projection points are (Dx, Dy), Dx is obtained by the following equation (2) from Dy = f. be able to.
Dx = (Ox / Oy) * f (2)
Further, by performing the same calculation on the yz plane, the z coordinate of the projection point can be obtained. By this calculation, a projection point can be represented on the zx plane with the center of the screen as the origin. Thereafter, the standard flow corresponding to the input image can be obtained by converting the size of the imaging surface into the size of the actual input image.

<Arrangement and movement of feature points without considering camera tilt>
When the tilt of the camera with respect to the horizontal plane is 0 degree, the feature point is placed on a straight line parallel to the y-axis placed at the height of the road surface reflecting the height of the camera set at the beginning. Move in parallel. A schematic diagram is shown in FIG. A point written as an environmental point in FIG. 4 is a feature point at this time. The environmental point will be described later.

<Arrangement of feature points considering camera tilt>
When the tilt of the camera with respect to the horizontal plane is φ, a point is placed on a plane tilted by φ with respect to the y-axis and moved along the plane. A schematic diagram is shown in FIG. A point written as a projected point in FIG. 5 is a feature point at this time. The projection point will be described later. However, tilting movement is assumed to accumulate errors due to repeated state changes such as curves and camera tilt. Therefore, feature points arranged in a reference environment in which the camera tilt is 0 degrees and the vehicle is in a straight traveling state and is less likely to cause an error are referred to as “environment points”. A feature point obtained by converting a coordinate position by a curve or camera tilt is referred to as a “projection point”.

  As shown in FIG. 5, the environmental point does not change in the z-axis direction, and accumulation of errors related to movement in the z-axis direction caused by tilting the camera can be prevented. Moreover, since it depends only on the movement in the y-axis direction, errors due to the movement every time can be prevented. The projection point is calculated from the position of the environment point, and the projection point is projected on the screen while maintaining the coordinates of the environment point. When the tilt of the camera is given, as shown in FIG. 5, the position of the environmental point is arranged in a state where it is inclined as much as the tilt of the camera.

<Projection point when turning>
Even when the camera rotates when the handle is turned off, the environmental point operates as it is. As the angle of the front wheel changes with respect to the longitudinal direction of the vehicle body by operating the steering wheel, when the camera rotates, the rotation radius and the center position of rotation corresponding to the angle are calculated. Based on this value and the environment point, the projection point is placed at the position of the curve. A schematic diagram of the arrangement of projection points is shown in FIG. The calculation of the arrangement of points at this time will be described later.
<Calculation of turning radius>
The turning radius of the vehicle can be obtained by the following equation (3).
r = Hcos θ + W / sin θ (3)
In this equation, r is the radius of rotation, H is the tread (the distance between the left and right wheels), W is the wheelbase (
The distance between the front wheel and the rear wheel), and θ is the inclination angle of the front wheel with respect to the longitudinal direction of the vehicle body. This formula (
FIG. 7 shows 3).
If the mobile body to be mounted is not a vehicle, the turning radius of the camera can be obtained by the following equation (3) ′.
r = v / Δθ (3) ′
In this equation, r is a rotation radius, v is a moving speed, and Δθ is a rotation angular speed.

<Placement of projection points during curve>
First, the center angle is calculated by making the distance in the y-axis direction from the camera to the environment point correspond to the length of the arc. In a circle whose radius is the distance from the center of rotation to the camera, the point moved from the camera position by the calculated central angle is the coordinate of the projection point corresponding to the environmental point. This is shown in FIG. In FIG. 8, s is a distance from the camera to the environment point, r is a radius of rotation, θ is a central angle, p is a position of the environment point, and p ′ is a position of the projection point. From FIG. 8, the value of the central angle θ can be obtained by the following equation.
θ = (s / 2πr) * 2π
= S / r

<Generation of normative flow>
The coordinates of the projection points on the camera image store the previous state, and a vector is generated by drawing the previous state as the flow start point and the current coordinate position as the flow end point. FIG. 9 shows an example of the generated normative flow. In this example, the normative flow OF1 when the vehicle is traveling straight is shown. Here, FOE in FIG. 9 indicates the FOE of the normative flow OF1.

(2) Feature Point Extraction and Flow Detection in Input Image Optical flow detection is performed by finding corresponding points between two consecutive input images. First, feature point extraction is performed on the first image based on Non-Patent Document 4. Then, a corresponding point with the second image is found to detect the optical flow. For example, Non-Patent Document 5 is used as the method for determining the corresponding points. FIG. 10 shows an example of a flow generated from an actual image. In this example, a flow OF2 generated from an object is included in addition to the reference flow OF1.

(3) Verification of normative flow and actual flow <Position flow field position correction>
When the reference flow generated by the reference flow generation process is compared with the optical flow detected from the image acquired by the actual imaging means, it is necessary to correct the positional deviation. For example, the deviation can be corrected by calculating the FOE (see FIG. 9) from the flow information detected from the actual input image and aligning the position with the FOE from the reference flow information. .
<Screen division and normative flow settings>
In this proposal, in order to detect an object by comparing the normative flow at each location and the optical flow of the image acquired by the imaging means, the location is associated with the normative flow. As a flow comparison method, a method for dividing a screen into blocks and setting a representative normative flow for each block will be described below.

<Representative normative flow settings>
In a straight-ahead state, a stationary object that is usually higher than the movement reference plane such as a road surface has a large flow in the screen because the distance from the camera installed on the vehicle is closer than the movement reference plane. Become. Since the normative flow is obtained by calculating the flow on the movement reference plane, the flow of the object should have a larger flow length than the normative flow. Therefore, the flow larger than the reference flow is considered as the flow of the object. At this time, when a plurality of normative flows are included in each block, the normative flow having the largest vector is represented. If a small vector is selected, a larger reference flow exists in the block, and even the flow of the moving reference plane without an object may exceed the representative vector. The normative flow appearing on the movement reference plane is considered to be almost the same vector in the local region, and is considered to be smoothly changing in the neighboring region. From this, when the reference flow is not included in the block, the vector value is taken from the neighboring block.

(4) Object detection <Determination of object flow>
All the optical flows in the image acquired by the imaging means are compared with the value of the representative standard flow of the block in which the flow exists. The comparison may be, for example, detecting an object flow having the same direction of the vertical component and the horizontal component and a long vector length as compared to the reference flow.
<Detection of object area>
However, if it is determined that a block including an object flow is an object region as it is, it may be determined that noise is also an object flow. Therefore, the following processing is performed. The feature points are extracted in a concentrated manner in the textured object region. That is, it is considered that the flow of a large number of objects is included in the block including the objects. On the other hand, in the case of noise, since it does not concentrate in a certain area, it is considered that a small number of noises determined to be an object flow are included in the block. Therefore, threshold processing is performed on the number of flows of objects included in the block, and it is determined that an object is included only when the number of object flows in the block exceeds the threshold. The block determined to contain the object may be displayed on the display unit 7 by changing the color, for example, as shown in FIG.

<Another screen division method>
Further, in dividing the area, it is used that the normative flow is not generated above the FOE of the imaging system. Further, since the normative flow and the object flow are compared, for example, a block is arranged at a place where the normative flow is generated and the comparison is performed. When a block is arranged in a part where the reference flow is generated, the start point and the end point obtained when the reference flow is obtained are arranged so as to be the upper end and the lower end of the block. FIG. 12 shows an example of equal area division. On the other hand, in FIG. 13, the reference flow is arranged so that the start point and the end point are diagonal positions of the block. However, when the imaging system performs a straight movement, the value of the horizontal component of the normative flow may be small and the block may be small. In order to prevent this, as shown in FIG. 14, the horizontal width may be a fixed width (constant), and the center of gravity of the block may be arranged so as to overlap the midpoint of the reference flow.

  The present invention has an effect that a stationary object can be accurately detected, and can be used when mounted not only on a vehicle but also on various moving bodies such as a mobile robot and a person.

It is a block diagram which shows one Embodiment of the object detection apparatus which concerns on this invention. It is a flowchart for demonstrating the process of the object detection apparatus of FIG. It is explanatory drawing which shows the screen projection calculation at the time of reference | standard flow production | generation. It is explanatory drawing which shows the environmental point setting at the time of reference | standard flow production | generation. It is explanatory drawing which shows projection point arrangement | positioning in case the camera inclines at the time of reference | standard flow production | generation. It is explanatory drawing which shows the projection point arrangement | positioning at the time of a curve at the time of reference | standard flow production | generation. It is explanatory drawing which shows the turning radius calculation at the time of a curve at the time of reference | standard flow production | generation. It is explanatory drawing which shows the center angle calculation for the projection point movement at the time of a curve at the time of reference | standard flow production | generation. It is explanatory drawing which shows an example of a normative flow. It is explanatory drawing which shows an example of the flow produced | generated from the real image. It is explanatory drawing which shows the example of a display of the detection image of an object. It is explanatory drawing which shows the example of equal area | region division. It is explanatory drawing which shows the example of the area | region division | segmentation used as the diagonal position of a block with the starting point and end point of a reference | standard flow. It is explanatory drawing which shows the example of the area | region division | segmentation which makes the horizontal width | variety fixed width | variety (constant) and the gravity center of a block overlaps the midpoint of a normative flow.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Vehicle information acquisition means 2 Normative flow production | generation means (1st optical flow production | generation means)
3 Image capturing means 4 Observation flow generating means (second optical flow generating means)
5 comparison verification unit (object detection means)
6 Peripheral object information constituent means 7 Display section

Claims (6)

In an object detection apparatus for detecting an object based on an image captured by an imaging means installed on a moving body,
A first optical for generating a first optical flow of a screen assumed in a standard environment where no object exists based on an installation state of the imaging unit installed on the mobile body and a movement state of the mobile body Flow generation means;
Second optical flow generation means for generating a second optical flow based on an image captured by the imaging means;
Object detection means for detecting an object based on the difference between the generated first and second optical flows;
An object detection apparatus characterized by comprising. The first optical flow generation means includes:
By setting the height from the reference plane of the imaging means, the focal length of the imaging means, and the inclination of the imaging means with respect to the horizontal plane as parameters, the feature points are arranged on the reference plane in the virtual space,
Acquiring the moving speed and rotational angular velocity of the imaging means at predetermined intervals, changing the coordinates of the feature points, and generating the previous feature point and the current feature point as the start point and end point of the first optical flow, respectively; The object detection apparatus according to claim 1. The moving body is a vehicle,
The first optical flow generation means includes:
The height of the imaging means from the road surface, the focal length of the imaging means, the inclination of the imaging means with respect to the horizontal plane, the distance between the front and rear wheels of the vehicle, and the distance between the right and left wheels of the vehicle. By setting the five fixed parameters, feature points are placed on the road surface in the virtual space,
The moving speed of the vehicle and the inclination angle of the front wheel with respect to the longitudinal direction of the vehicle body are received as variable parameters at predetermined time intervals, the coordinates of the feature points are changed, and the previous feature point and the current feature point are respectively changed to the first optical point. The object detection device according to claim 1, wherein the object detection device is generated as a start point and an end point of a flow.   The object detection unit divides the screen imaged by the imaging unit into regions, sets a representative flow of the first optical flow for each region, and among the generated second optical flows, The object detection apparatus according to claim 1, wherein a flow larger than the representative flow is determined as an object.   The object detection means sets the divided area using the screen imaged by the imaging means and the information of the start point and end point of the optical flow obtained when the first optical flow is generated. The object detection apparatus according to claim 4.   6. The object detection device according to claim 3, wherein the object detection unit determines that a block whose number of flows determined to be the object exceeds a threshold value is an object block. 7.

Images