JP2014038401A - Vehicle periphery monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a vehicle periphery monitoring device capable of accurately detecting the existence of an object moving around a vehicle even when the posture of the vehicle abruptly changes.SOLUTION: A vehicle periphery monitoring device comprises: a change amount calculation unit (44) that calculates a translation vector (upward arrow t) and a rotation matrix (R) representing a change amount of the position and posture of an imaging unit (14); and vehicle movement component cancellation means (48) that cancels a movement component of a vehicle (12) between a first captured image (Ia) and a second captured image (Ib) on the basis of a plane projection matrix (H) calculated from the translation vector (upward arrow t) and the rotation matrix (R).

Description

  The present invention is a vehicle that detects an object existing around a vehicle based on a plurality of captured images obtained by capturing images at different times using an imaging unit mounted on the vehicle while the vehicle is traveling. The present invention relates to a peripheral monitoring device.

  2. Description of the Related Art Conventionally, a technique for detecting an object existing outside a vehicle or monitoring a relative positional relationship between the object and the host vehicle is known for the purpose of driving support of the vehicle. In particular, various methods have been proposed for imaging using an in-vehicle camera (imaging unit) mounted in front of a vehicle and detecting an object based on the obtained captured image.

  This in-vehicle camera is fixed and installed in any part of the vehicle. Therefore, for example, the position of the object in the captured image of the in-vehicle camera varies with the variation of the vehicle pitch angle (the vehicle inclination angle around the pitch axis that is the axis in the vehicle width direction). And this fluctuation | variation becomes an error factor in grasping | ascertaining the relative positional relationship of the object which exists outside the vehicle, and the own vehicle.

  On the other hand, the position detection accuracy can be improved by correcting the position of the object recognized from the captured image in accordance with the estimated value of the pitch angle of the vehicle. For example, Patent Document 1 proposes a method for estimating the pitch angle of a vehicle.

Japanese Patent No. 3808287

  The present invention has been made in connection with the technical idea disclosed in the above-mentioned Patent Document 1. Even when the attitude of the vehicle suddenly changes, the presence of an object that moves around the vehicle is detected. An object of the present invention is to provide a vehicle periphery monitoring device that can be accurately detected.

  The vehicle periphery monitoring apparatus according to the present invention is mounted on a vehicle, and an imaging unit that acquires a plurality of captured images by capturing images at different times during traveling of the vehicle, and a first captured image at the first imaging time point. A translation vector and a rotation matrix that represent the amount of change in the position / orientation of the imaging unit when the second captured image is captured at the second imaging time point with reference to the position / orientation of the imaging unit at the time of imaging. Based on the translation vector and the rotation matrix calculated by the change amount calculation unit, and a second amount indicated by the second picked-up image from the first image pickup surface indicated by the first picked-up image. Based on the plane projection matrix calculated by the conversion matrix calculation unit, a conversion matrix calculation unit that calculates a plane projection matrix that represents the projection conversion to the imaging surface, the first captured image and the second captured image Cancels out the motion component of the vehicle between Characterized in that it comprises a vehicle motion component canceling means.

  As described above, based on the change amount calculation unit for calculating the translation vector and the rotation matrix representing the change amount of the position and orientation of the image pickup unit, and the plane projection matrix calculated from the translation vector and the rotation matrix, the first image pickup Since the vehicle motion component canceling means for canceling the motion component of the vehicle between the image and the second captured image is provided, the amount of change in the position and orientation of the imaging unit caused by the change in the posture of the vehicle Equal vehicle motion components can be offset. Thereby, even if the posture of the vehicle suddenly changes, the presence of an object moving around the vehicle can be accurately detected.

  Moreover, it is preferable that the said conversion matrix calculation part calculates the said plane projection matrix showing the projective transformation from the road area | region included in the said 1st captured image to the road area | region included in the said 2nd captured image. . Thereby, the detection / recognition accuracy of the object existing on the road area (motion component canceling effect) can be enhanced. An object present on the road area is particularly preferable because it has a higher risk of collision with the vehicle. On the other hand, since an object existing at a position away from the road area cannot be correctly detected as a target object, a troublesome over-alarm or the like can be suppressed.

  Furthermore, it is preferable that the vehicle motion component canceling unit is an image creating unit that creates an image in which the motion component of the vehicle is canceled.

  Furthermore, it is preferable to further include a moving object detection unit that detects an object moving around the vehicle based on at least one of the images generated by the image generation unit.

  Furthermore, it is preferable that the moving object detection unit detects the object based on a plurality of the images in which at least one of the first imaging time point and the second imaging time point is different.

  Furthermore, it is preferable that the image creation unit creates a difference image as the image. In this way, by applying the image difference method, the amount of calculation processing can be significantly reduced as compared with other image processing methods.

  Further, the image creation unit extracts a first region of interest including the object from the first captured image, and extracts a second region of interest including the object from the second captured image. Preferably, the image between the first region of interest and the second region of interest is created.

  Furthermore, the image creation unit identifies the position of the second region of interest as a position obtained by performing orthographic transformation based on the planar projection matrix with respect to the first reference position that identifies the position of the first region of interest. It is preferable to create the image as the second reference position.

  In addition, the image creation unit identifies the position of the first region of interest as a position obtained by performing reverse projection transformation based on the planar projection matrix with respect to the second reference position that identifies the position of the second region of interest. It is preferable to create the image as the first reference position.

  Furthermore, the image creation unit includes the image between the captured image obtained by performing orthographic transformation based on the planar projection matrix with respect to the first captured image and the second captured image, and the second captured image. Preferably, any one of the images between the captured image that has been subjected to reverse projection transformation based on the planar projection matrix and the first captured image is created. Thereby, it is possible to create an image taking into account the change in the form of the object on the captured image accompanying the change in the position and orientation of the imaging unit.

  According to the vehicle periphery monitoring apparatus according to the present invention, a change amount calculation unit that calculates a translation vector and a rotation matrix that represents a change amount of the position and orientation of the imaging unit, and a planar projection calculated from the translation vector and the rotation matrix Since the vehicle motion component canceling unit that cancels the motion component of the vehicle between the first captured image and the second captured image is provided based on the matrix, the imaging unit is caused due to a change in the posture of the vehicle. The motion component of the vehicle equal to the amount of change in the position / posture can be offset. Thereby, even if the posture of the vehicle suddenly changes, the presence of an object moving around the vehicle can be accurately detected.

It is a block diagram which shows the structure of the vehicle periphery monitoring apparatus which concerns on this Embodiment. It is a schematic perspective view of the vehicle carrying the vehicle periphery monitoring apparatus shown in FIG. 2 is a flowchart provided for explaining the operation of the ECU shown in FIG. 4A and 4B are diagrams illustrating captured images acquired by imaging using a camera. FIG. 5A is a schematic explanatory diagram illustrating a geometric positional relationship among a camera, a human body, and a road surface. FIG. 5B is a schematic explanatory diagram illustrating a geometric positional relationship between the camera and the road surface at the time of the first imaging. FIG. 6A is a schematic explanatory diagram illustrating an example of extraction of pedestrians in the first captured image. FIG. 6B is a schematic explanatory diagram of a method of setting a region of interest in the second captured image when changes in the position / posture of the camera are not considered. 7A and 7B are schematic explanatory diagrams of a method for setting a region of interest in the second captured image in consideration of changes in the position and orientation of the camera. 8A and 8B are schematic explanatory diagrams regarding the position and height of a pedestrian in each imaging region. It is a schematic explanatory drawing regarding the production method of a difference image. It is a schematic block diagram which shows an example of the production method of a difference image and a motion detection image. FIG. 11A and FIG. 11B are schematic explanatory diagrams for explaining the operational effects obtained by this detection processing.

  Hereinafter, preferred embodiments of a vehicle periphery monitoring device according to the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing a configuration of a vehicle periphery monitoring apparatus 10 according to the present embodiment. FIG. 2 is a schematic perspective view of the vehicle 12 on which the vehicle periphery monitoring device 10 shown in FIG. 1 is mounted.

  As shown in FIGS. 1 and 2, the vehicle periphery monitoring device 10 is a color camera (hereinafter simply referred to as “camera 14”) that captures a color image (hereinafter referred to as a captured image) including a plurality of color channels. The vehicle speed sensor 16 for detecting the vehicle speed Vs of the vehicle 12, the yaw rate sensor 18 for detecting the yaw rate Yr of the vehicle 12, the brake sensor 20 for detecting the brake pedal operation amount Br by the driver, and the vehicle periphery monitoring device 10 An electronic control device (hereinafter referred to as “ECU 22”), a speaker 24 for issuing an alarm or the like by sound, and a display device 26 for displaying a captured image output from the camera 14.

  The camera 14 is a camera that mainly uses light having a wavelength in the visible light region, and functions as an imaging unit that images the periphery of the vehicle 12. The camera 14 has a characteristic that the output signal level increases as the amount of light reflected on the surface of the subject increases, and the luminance (for example, RGB value) of the image increases. As shown in FIG. 2, the camera 14 is fixedly disposed (mounted) at a substantially central portion of the front bumper portion of the vehicle 12.

  The imaging means for imaging the periphery of the vehicle 12 is not limited to the above configuration example (so-called monocular camera), and may be, for example, a compound eye camera (stereo camera). Further, an infrared camera may be used instead of the color camera, or both may be provided. Further, in the case of a monocular camera, another ranging means (radar apparatus) may be provided.

  Returning to FIG. 1, the speaker 24 outputs an alarm sound or the like in response to a command from the ECU 22. The speaker 24 is provided on a dashboard (not shown) of the vehicle 12. Alternatively, instead of the speaker 24, an audio output function provided in another device (for example, an audio device or a navigation device) may be used.

  The display device 26 (see FIGS. 1 and 2) is a HUD (head-up display) disposed on the front windshield of the vehicle 12 at a position that does not obstruct the driver's front view. The display device 26 is not limited to the HUD, but a display that displays a map or the like of a navigation system mounted on the vehicle 12 or a display (MID; multi-information display) that displays fuel consumption or the like provided in a meter unit or the like is used. can do.

  The ECU 22 basically includes an input / output unit 28, a calculation unit 30, a display control unit 32, and a storage unit 34.

  Signals from the camera 14, the vehicle speed sensor 16, the yaw rate sensor 18, and the brake sensor 20 are input to the ECU 22 side via the input / output unit 28. Each signal from the ECU 22 is output to the speaker 24 and the display device 26 via the input / output unit 28. The input / output unit 28 includes an A / D conversion circuit (not shown) that converts an input analog signal into a digital signal.

  The calculation unit 30 performs calculations based on the signals from the camera 14, the vehicle speed sensor 16, the yaw rate sensor 18, and the brake sensor 20, and generates signals for the speaker 24 and the display device 26 based on the calculation results. The calculation unit 30 includes an object extraction unit 40, a plane parameter estimation unit 42, a change amount calculation unit 44, a transformation matrix calculation unit 46, a difference image creation unit 48 (vehicle motion component offset means, an image creation unit), and a moving object detection. Functions as the unit 50 and the motion detection unit 52. The function of each unit is realized by reading and executing a program stored in the storage unit 34. Alternatively, the program may be supplied from the outside via a wireless communication device (mobile phone, smartphone, etc.) not shown.

  The display control unit 32 is a control circuit that drives and controls the display device 26. The display control unit 32 drives the display device 26 by outputting a signal used for display control to the display device 26 via the input / output unit 28. As a result, the display device 26 can display various images (captured images, marks, etc.).

  The storage unit 34 includes an imaging signal converted into a digital signal, a RAM (Random Access Memory) that stores temporary data used for various arithmetic processes, and a ROM (Read Only Memory) that stores an execution program, a table, a map, or the like. ) Etc.

  The vehicle periphery monitoring apparatus 10 according to the present embodiment is basically configured as described above. An outline of the operation of the vehicle periphery monitoring device 10 will be described below.

  The ECU 22 converts the analog video signal output from the camera 14 into a digital signal at predetermined frame clock intervals / cycles (for example, 30 frames per second) and temporarily captures it in the storage unit 34. Then, the ECU 22 performs various arithmetic processes on the captured image (the front image of the vehicle 12) read from the storage unit 34.

  The ECU 22 (especially the calculation unit 30) comprehensively considers the processing result for the captured image and each signal (vehicle speed Vs, yaw rate Yr, and operation amount Br) indicating the traveling state of the vehicle 12 as necessary, and the front of the vehicle 12 Are detected as objects to be monitored (hereinafter referred to as “monitoring objects” or simply “objects”).

  When the calculation unit 30 determines that there is a high possibility that the vehicle 12 is in contact with the monitoring target, the ECU 22 controls each output unit of the vehicle periphery monitoring device 10 in order to call the driver's attention. For example, the ECU 22 outputs an alarm sound (for example, a beeping sound) via the speaker 24 and highlights the part of the monitored object in the captured image visualized on the display device 26. Let

  Next, the detailed operation of the vehicle periphery monitoring device 10 will be described with reference to the flowchart of FIG. Note that this processing flow is executed for each imaging frame when the vehicle 12 is traveling.

  In step S1, the ECU 22 acquires a captured image that is an output signal in front of the vehicle 12 (predetermined angle of view range) captured by the camera 14 for each frame.

  As shown in FIG. 4A, it is assumed that a captured image of the first frame (hereinafter referred to as a first captured image Ia) is obtained at the first imaging time point T1. Then, the ECU 22 temporarily stores the acquired first captured image Ia in the storage unit 34. For example, when an RGB camera is used as the camera 14, the obtained captured image is a multi-gradation image composed of three color channels.

  In step S2, the calculation unit 30 determines whether two or more captured images have already been acquired. When it is determined that it has been acquired (step S2: YES), the process proceeds to step S5 described later. On the other hand, when it is determined that it has not been acquired (step S2: NO), the process proceeds to the next step (S3).

  In step S3, the object extraction unit 40 extracts the object from the first captured image Ia acquired in step S1. In this extraction, various known image recognition methods (including pattern matching) applicable to still images may be used. In addition to the above-described processing, the object extraction unit 40 determines a position (hereinafter referred to as a reference position) for specifying the presence of each extracted object, and temporarily stores it in the storage unit 34. Remember me.

  In the example of FIG. 4A, in the first captured image Ia, a road area (hereinafter simply referred to as “road 60a”) on which the vehicle 12 travels, and a plurality of utility pole areas (hereinafter simply referred to as “ There is a pedestrian area (hereinafter simply referred to as “pedestrian 64a”) on the road 60a. For example, it is assumed that the object extraction unit 40 extracts a road 60a and a pedestrian 64a.

  In step S4, after waiting for the frame interval time, the ECU 22 proceeds to the next imaging operation (step S1) at the second imaging time point T2.

  As shown in FIG. 4B, it is assumed that a captured image of the second frame (hereinafter referred to as a second captured image Ib) is obtained. In the second captured image Ib, as in the case of the first captured image Ia, a road area (hereinafter, road 60b), a plurality of utility pole areas (hereinafter, telephone pole 62b), and a pedestrian area (hereinafter, pedestrian 64b) are included. Each exists. However, as the frame interval time elapses, the relative positional relationship between the camera 14 and each object changes every moment. Thereby, even if the 1st captured image Ia and the 2nd captured image Ib are the same view angle range, each target object is imaged with a respectively different form (shape, magnitude | size, or color).

  In step S2 again, the calculation unit 30 determines that two or more captured images have been acquired (step S2: YES), and proceeds to the next step (S5).

  In step S5, the plane parameter estimation unit 42 estimates a plane parameter representing the road 60a (actual road surface) detected in step S3. Here, various known image processing techniques can be used for estimating the plane parameters.

  As shown in FIG. 5A, the geometric positional relationship between the camera 14 and the actual road surface (hereinafter simply referred to as the road surface S) is specified by the normal vector ↑ n as the plane parameter and the distance d. Here, the normal vector ↑ n is a unit vector along the normal direction of the road surface S. The distance d is a distance between the optical center C of the camera 14 and the road surface S when the first captured image Ia is captured.

By the way, the road surface S is generally substantially parallel to the vehicle width direction of the vehicle 12, and is therefore substantially parallel to the lateral direction of the image captured by the camera 14 (X-axis direction of the camera coordinate system). Therefore, the normal vector ↑ n on the first camera coordinate system is expressed as ↑ n = [0, cos θ, sin θ] ^T using the inclination angle θ of the road surface S with respect to the optical axis Lc of the camera 14.

  In step S <b> 6, the change amount calculation unit 44 calculates the change amount of the position / orientation of the camera 14. Specifically, the change amount calculation unit 44 uses the position / posture of the camera 14 when the first captured image Ia is captured as a reference for the position / position of the camera 14 when the second captured image Ib is captured. The amount of change in posture is calculated. Here, the change amount calculation unit 44 calculates the change amount at the later imaging time point (T2) with reference to the earlier imaging time point (T1), but the reverse may be possible. That is, the change amount calculation unit 44 may calculate the change amount at the earlier imaging time point (T1) with reference to the later imaging time point (T2).

  The change amount calculation unit 44 uses a known technique such as SfM (Structure from Motion) or the like to change the spatial position change amount of the camera 14 (specifically, from the first captured image Ia and the second captured image Ib). , A translation vector ↑ t) is calculated. As shown in FIG. 5B, the translation vector ↑ t is a vector having a reference point of the camera 14 at the first imaging time point T1 as a starting point and an end point being the reference point of the camera 14 at the second imaging time point T2. The reference point is a point fixed with respect to the camera 14 and corresponds to, for example, the optical center C of the camera 14.

  Further, the change amount calculation unit 44 calculates a change amount (specifically, a rotation matrix R) of the spatial orientation of the camera 14 from the first captured image Ia and the second captured image Ib. As shown in FIG. 5B, when the origin of the camera coordinate system at the first imaging time point T1 is made coincident with the origin of the camera coordinate system at the second imaging time point T2, as shown in FIG. It is a matrix expressing coordinate transformation between them (rotational transformation around three axes). The camera coordinate system is a coordinate system fixed with respect to the camera 14, and a coordinate system fixed with respect to the camera 14 with the reference point (optical center C) as the origin (three-axis coordinate system) as in this example. ). In the present embodiment, the camera coordinate system is such that the optical axis direction of the camera 14 is the Z-axis direction, the lateral direction (vehicle width direction) of the camera 14 is the X-axis direction, and the vertical direction of the camera 14 (Z-axis direction and X-axis). This is a three-axis coordinate system in which the direction perpendicular to the direction) is the Y-axis direction.

  When calculating the translation vector ↑ t and the rotation matrix R, specifically, a technique (for example, “Structure from Motion without Correspondence” in which optical flow estimation between the first captured image Ia and the second captured image Ib is applied to SfM. / Frank Dellaert, Steven M. Seitz, Charles E. Thorpe, Sebastian Thrun / see Computer Sience Department & Robotics Institute Canegie Mellon University, Pittsborgh PA 15213). Further, in order to calculate the translation vector ↑ t and the rotation matrix R, three or more captured images may be used.

  In step S7, the transformation matrix calculation unit 46 uses the plane parameter estimated in step S5 and the translation vector ↑ t and rotation matrix R calculated in step S6, respectively, to generate a plane projection matrix H (homography matrix). Is calculated. Here, the planar projection matrix H is a two-dimensional matrix that represents planar projection conversion from the first imaging plane (see FIG. 5B) to the second imaging plane (see the figure).

  The transformation matrix calculation unit 46 may calculate the planar projection matrix H by a combination of a known method and an arbitrary constraint condition. In the present embodiment, the planar projection matrix H corresponds to a matrix that represents the projective transformation from the road 60a included in the first captured image Ia to the road 60b included in the second captured image Ib.

In this case, the relationship between the position vector ↑ P1 of the projection point P1 in the first captured image Ia and the position vector ↑ P2 of the projection point P2 in the second captured image Ib is associated by plane projective transformation. Specifically, the plane projection matrix H for performing the plane projection transformation is expressed by the following equation (1) using the above-described translation vector ↑ t, rotation matrix R, and plane parameters (↑ n, d) of the road surface S. Given by.
H = R ^T −R ^T · ↑ t · (↑ n / d) ^T (1)

The position vector ↑ P2 is given by the following expression (2) using the position vector ↑ P1 and the planar projection matrix H. Note that k is a positive proportionality constant.
↑ P2 = k ・ H ・ ↑ P1 (2)

Accordingly, the position vector ↑ P2 is expressed by the following equation (3) by substituting the planar projection matrix H of the equation (1) into the equation (2).
↑ P2 = k · (R ^T −R ^T · ↑ t · (↑ n / d) ^T ) · ↑ P1 (3)

  In step S8, the difference image creation unit 48 creates a difference image D between the first captured image Ia and the second captured image Ib by using the planar projection matrix H calculated in step S7. Prior to the creation of the difference image D, the difference image creation unit 48 includes an image region (hereinafter referred to as a first region of interest) including the pedestrian 64a corresponding to the human body M (see FIG. 5B) from the first captured image Ia. ) Is set. Moreover, the difference image creation part 48 sets the image area | region (henceforth a 2nd region of interest) containing the pedestrian 64b corresponding to the human body M (refer FIG. 5B) from the 2nd captured image Ib.

  FIG. 6A is a schematic explanatory diagram illustrating an extraction example of the pedestrian 64a in the first captured image Ia. In step S3 described above, it is assumed that the object extraction unit 40 determines the contact position between the road 60a and the pedestrian 64a (that is, the contact point between the road 60a and the pedestrian 64a) as the first reference position 70. For example, it is assumed that the differential image creation unit 48 sets the first region of interest 72 having a rectangular shape according to a predetermined rule with the first reference position 70 as a mark. Then, as shown in the figure, in the first captured image Ia, the first region of interest 72 is set so as to include all the pedestrians 64a illustrated by broken lines.

  However, as shown in FIG. 6B, when a region of interest (first region of interest 72) having the same position and size as in FIG. 6A is set for the second captured image Ib, the tip portions of both feet of the pedestrian 64b are extracted. become unable. This is because an unexpected displacement between the pedestrians 64a and 64b cannot be followed by the sudden change (for example, pitching) of the posture of the vehicle 12. As a result, the presence or state of the pedestrians 64a and 64b may not be accurately grasped based on the obtained difference image D.

  Therefore, in the present embodiment, the difference image D in which the amount of change in the position / orientation of the camera 14 is offset is created using the already calculated plane projection example H. Here, the difference image creation unit 48 uses the planar projection example H to determine the first reference position 70 of each object (specifically, the pedestrian 64a) in the first captured image Ia as the second captured image. It converts into each 2nd reference position 74 of each target object (specifically pedestrian 64b) in Ib.

  The difference image creation unit 48 converts the first reference position 70 corresponding to the first captured image Ia to the second reference position 74 corresponding to the second captured image Ib according to the above-described equation (3). Then, the difference image creation unit 48 uses the second reference position 74 as a mark, takes into account the change in appearance due to the position change, the moving component of the object, and the like, and then takes the rectangular second interest according to the same rules as in FIG. 6A. Region 76 is set.

  As a result, as shown in FIG. 7B, the second region of interest 76 can be set to include all the pedestrians 64b in the second captured image Ib. Thereby, even if it is a case where the attitude | position of the vehicle 12 changes suddenly, the detection precision of the pedestrian 64a (64b) from the obtained difference image D improves.

Note that the first reference position 70 corresponding to the first captured image Ia may be converted not only to the second reference position 74 corresponding to the second captured image Ib, but may also be subjected to inverse conversion, that is, reverse projection conversion. . In this case, it is assumed that the second reference position 74 in the second captured image Ib is known. The relationship between the position vector ↑ P2 of the projection point P2 in the second captured image Ib and the position vector ↑ P1 of the projection point P1 in the first captured image Ia can be expressed as follows by modifying equation (2). (4).
↑ P1 = (1 / k) · H ⁻¹ · ↑ P2 (4)

Accordingly, the position vector ↑ P1 is expressed by the following equation (5) by substituting the planar projection matrix H of the equation (1) into the equation (4). Here, the subscript “−1” represents an inverse matrix.
↑ P1 = (1 / k) · (R ^T −R ^T · ↑ t · (↑ n / d) ^T ) ⁻¹ · ↑ P2 (5)

  Further, the difference image creation unit 48 maps one of the first captured image Ia and the second captured image Ib (all or a part) according to the planar projection matrix H (in the case of the first captured image Ia). May be subjected to image difference processing after normal projection conversion, or reverse projection conversion in the case of the second captured image Ib. Thereby, the difference image D can be created taking into consideration the change in the form of the object (pedestrians 64a and 64b) on the image accompanying the change in the position and orientation of the camera 14.

  Then, the difference image creation unit 48 normalizes the sizes of the first region of interest 72 and the second region of interest 76 that are subjected to the difference processing. Accordingly, a first normalized image corresponding to the first region of interest 72 and a second normalized image corresponding to the second region of interest 76 are created.

  As shown in FIG. 9, the height of the pedestrian 64a indicated by the first normalized image 80a substantially matches the height of the pedestrian 64b indicated by the second normalized image 80b. By performing a difference calculation on the first normalized image 80a and the second normalized image 80b having the same size, a difference image D in which the difference area between the two is imaged is obtained. In order to improve the detection accuracy of the difference area, it is preferable to adjust the relative positional relationship between the first normalized image 80a and the second normalized image 80b before the difference calculation. In this example, both hands 84 and both feet 86 of the pedestrian 64b are extracted as the different areas.

  As described above, the difference image creating unit 48 is based on the planar projection matrix H, and the position / posture of the camera 14 between the first captured image Ia and the second captured image Ib (that is, the motion component of the vehicle 12). This is one form of vehicle motion component canceling means for canceling The target to which the cancellation calculation is performed is not limited to the captured image, but may be an object extracted from the captured image, a feature amount, or the like.

  In step S9, the moving object detection unit 50 detects each object that moves around the vehicle 12 based on at least one difference image D created in step S8. Here, when a plurality of (two or more) difference images D are used, at least one of the first imaging time T1 and the second imaging time T2 needs to be different. For example, the moving object detection unit 50 detects the presence of the human body M at a predetermined position on the road surface S from the characteristics of the extracted components (both hands 84 and both feet 86) in the difference image D in FIG.

  In step S10, the motion detection unit 52 detects the motion of each object detected in step S9. For example, the motion detection unit 52 may estimate the moving amount and moving direction of the human body M from the distance between the vehicle 12 and the human body M and the displacement amount of the pedestrians 64a and 64b between the frames. Further, the motion detection unit 52 may detect that the human body M is walking from the characteristics of the extracted components (both hands 84 and both feet 86) in the difference image D in FIG.

  In step S11, the target object extraction unit 40 updates each target object as an extraction target (monitoring target). For example, the object extraction unit 40 adds a new object that has entered the imaging area of the camera 14 and excludes an object that has advanced outside the imaging area. Further, when it is determined that the coincidence between the first normalized image 80a and the second normalized image 80b is lower than a predetermined threshold value, the object may be excluded from being monitored.

  In step S12, the object extraction unit 40 updates the reference position of each object updated in step S10. In the example of the pedestrian 64a (human body M) shown in FIGS. 8A and 8B, the object extraction unit 40 updates and sets the reference position corresponding to the human body M from the first reference position 70 to the second reference position 74. .

  In step S13, the ECU 22 causes the storage unit 34 to store data necessary for the next calculation process. For example, the difference image D created in step S8, the attribute of the extraction target updated in step S11, the reference position updated in step S12, and the like can be mentioned.

  In this way, the ECU 22 operates according to steps S1 to S13 for each frame of imaging. In the above description, the difference processing under the relationship between the first frame and the second frame has been described, but the operation when expanding to the nth frame (n is a natural number) will be described with reference to FIG. Here, the captured image of the nth frame is denoted as I (n).

  As illustrated in FIG. 10, after obtaining the current captured image I (n + 1) in the (n + 1) th frame, the transformation matrix calculation unit 46 calculates the current captured image I from the most recent captured image I (n). A planar projection matrix H (n) that maps to (n + 1) is created. Then, the difference image creation unit 48 calculates the difference image D between the captured image I (n + 1) and the captured image I (n) obtained by performing orthogonal projection transformation based on the planar projection matrix H (n). (N + 1) is created. By the way, in the nth frame that is the previous frame, the difference image creation unit 48 has already created the difference image D (n) in the same manner as described above.

  Thereafter, the moving object detection unit 50 creates a motion detection image Im (n) by performing a predetermined operation, for example, an AND operation, on the two difference images D (n + 1) and D (n). By referring to the motion detection image Im (n), it is possible to detect the presence of an object moving between three frames {(n-1) th, nth, and (n + 1) th frames}. . In addition, the motion detection unit 52 detects the motion of each object based on the image information and the position information obtained in the process of creating the difference image D (n) and the motion detection image Im (n).

  By sequentially executing this operation, the vehicle periphery monitoring apparatus 10 can monitor an object (for example, the human body M in FIG. 5B) existing in front of the vehicle 12 at a predetermined time interval. By the way, in addition to the effect of offsetting the amount of change in the position and orientation of the camera 14 by applying the detection process according to the present embodiment (hereinafter referred to as the main detection process), there are other effects related to the detection accuracy of the object. can get. Hereinafter, the effect will be described with reference to FIGS. 11A and 11B.

  The plane projection matrix H shown in the above-described equation (1) is a matrix representing the projection transformation of each projection point existing on the road surface S. That is, the planar projection matrix H is a matrix suitable for correcting the position / posture of an object existing on the road surface S. On the other hand, the planar projection matrix H is not necessarily suitable for correcting the position / posture of an object that is three-dimensionally separated from the road surface S (the vehicle width direction, the depth direction, and the height direction). That is, the position conversion accuracy based on the planar projection matrix H is higher as the object is closer to the road surface S and lower as the object is farther from the road surface S.

  FIG. 11A is a schematic explanatory diagram illustrating an example of projective transformation of the first region of interest 100 in the first captured image Ia. A traffic sign (not shown) belonging to an object not to be monitored has a circular sign plate fixed to the end of a long pole, and stands on the road surface S (see FIG. 5B). In the lower right part of the first region of interest 100, there is a mark 102 marked on one side of the sign board.

  Here, as described with reference to FIGS. 6A to 7B, the difference image creation unit 48 performs orthographic transformation based on the planar projection matrix H on the first reference position 104 that specifies the first region of interest 100. , A second reference position 106 and a second region of interest 108 are obtained. However, a mark 110 different from the shape of the mark 102 exists at the lower right corner of the second region of interest 108 in the second captured image Ib. This is because the sign board is located far above the road surface S, which is the reference plane, and the position conversion accuracy in the vicinity of the mark 102 in the first captured image Ia is not high.

  As a result, the moving object detection unit 50 and the motion detection unit 52 detect that the sign board (marks 102 and 110) has changed greatly between the frames of the imaging. And the target object extraction part 40 can exclude this marker board from extraction object (monitoring object) noting that it is detection abnormality. In this way, by sequentially executing this detection process, an object (for example, a sign / signboard, etc.) that is likely to be erroneously detected during normal image recognition processing can be detected from the monitoring target without performing different processes. Can be excluded.

  FIG. 11B is another exemplary view of the first captured image Ia. The first captured image Ia includes a road area where the vehicle 12 travels (hereinafter simply “road 112”), a crossover area that crosses the road 112 (hereinafter simply “overpass 114”), and a pedestrian area existing on the road 112. (Hereinafter simply “road pedestrian 116”) and pedestrian areas existing on the overpass 114 (hereinafter simply “overpass pedestrian 118”).

  Considering in the same manner as in the example of FIG. 11A, regarding the road pedestrian 116 close to the point 120 on the road 112, the change in position between frames can be accurately obtained using the planar projection matrix H. Thereby, the moving object detection part 50 can detect the road pedestrian 116 as an object correctly using the image which canceled the motion component of the vehicle 12. FIG.

  On the other hand, with respect to the overpass pedestrian 118 far from the point 122 on the road 112, the position change between frames cannot be accurately obtained using the planar projection matrix H. Thereby, the moving target object detection part 50 can exclude the overpass pedestrian 118 from a target object.

  In this way, by using this detection process, the detection / recognition accuracy of an object (road pedestrian 116) existing on the road 112 can be improved. This is particularly preferable because an object present on the road 112 has a higher risk of collision with the vehicle 12. Conversely, an object with a low risk of collision with the vehicle 12, that is, an object existing at a position away from the road 112 (overpass pedestrian 118) cannot be correctly detected as an object, and thus annoying over-alarms and the like are suppressed. can do.

  By the way, this detection process functions effectively within the normally assumed operation range. However, there are cases where the detection accuracy of each object cannot be sufficiently secured by this detection process. Therefore, the calculation unit 30 can execute a plurality of detection processes (including the main detection process), and determines the suitability of the main detection process each time a captured image is acquired, and then executes one detection process. You may choose.

  As a first aptitude, it may be considered whether a road region can be extracted from a captured image. This is because the road 60a (the road surface S in FIG. 5B) in the first captured image Ia needs to be extracted for each frame in order to apply this detection process. For example, when the road 60a is obstructed from the imaging field of view of the camera 14, such as when another vehicle is present in front of the own vehicle 12, the plane parameter cannot be estimated with high accuracy, and the detection accuracy of each object is reduced. Probability is high. When detecting this state, the arithmetic unit 30 may use position information regarding an epipole, a flow (motion vector), and the like.

  As a second aptitude, it may be considered whether the calculation accuracy of the planar projection matrix H can be sufficiently secured. For example, when the traveling speed of the vehicle 12 is extremely low, a calculation error of the plane projection matrix H is likely to occur, and there is a high possibility that the detection accuracy of each target object is lowered. Therefore, the calculation unit 30 may determine whether or not the vehicle speed Vs acquired from the vehicle speed sensor 16 exceeds a predetermined threshold value.

  As described above, the vehicle periphery monitoring device 10 according to the present invention is mounted on the vehicle 12 and acquires a plurality of captured images Ia and Ib by capturing images at different imaging time points T1 and T2 while the vehicle 12 is traveling. 14 and when the second captured image Ib is captured at the second imaging time point T2 with reference to the position and orientation of the camera 14 when the first captured image Ia is captured at the first imaging time point T1. Based on the translation vector ↑ t and the rotation matrix R that represent the amount of change in the position and orientation of the camera 14 and the translation vector ↑ t and the rotation matrix R, the first captured image Ia indicates the first. Based on the plane projection matrix H, a conversion matrix calculation unit 46 that calculates a plane projection matrix H representing projection conversion from the imaging plane to the second imaging plane indicated by the second captured image Ib, and the first captured image Ia and the second The luck of the vehicle 12 between the captured image Ib A difference image creation unit 48 that creates a difference image D with components canceled out, and a moving object detection unit that detects an object (for example, a human body M) that moves around the vehicle 12 based on at least one difference image D. 50.

  As described above, the translation vector ↑ t representing the change amount of the position / orientation of the camera 14 and the change amount calculation unit 44 for calculating the rotation matrix R, and the plane projection matrix H calculated from the translation vector ↑ t and the rotation matrix R are used. Since the vehicle motion component canceling means (for example, the difference image creation unit 48) for canceling the motion component of the vehicle 12 between the first captured image Ia and the second captured image Ib is provided based on the attitude of the vehicle 12 It is possible to cancel the motion component of the vehicle 12 equal to the amount of change in the position / posture of the camera 14 caused by the fluctuation of the camera 14. Thereby, even if the posture of the vehicle 12 is suddenly changed, the presence of the object (human body M) that moves around the vehicle 12 can be accurately detected.

  Here, the image in which the change amount of the position / orientation of the camera 14 (the motion component of the vehicle 12) is canceled is not limited to the difference image D, and is an image showing an optical flow obtained by using a gradient method, a block matching method, or the like. There may be. In particular, by applying the image difference method, the amount of calculation processing can be significantly reduced as compared with other image processing methods.

  In addition, this invention is not limited to embodiment mentioned above, Of course, it can change freely in the range which does not deviate from the main point of this invention.

  In the present embodiment, the above-described image is created for the captured image obtained by the monocular camera (camera 14). However, it goes without saying that a similar effect can be obtained even with a compound-eye camera (stereo camera).

DESCRIPTION OF SYMBOLS 10 ... Vehicle periphery monitoring apparatus 12 ... Vehicle 14 ... Camera 22 ... ECU
DESCRIPTION OF SYMBOLS 28 ... Input / output part 30 ... Operation part 34 ... Storage part 40 ... Object extraction part 42 ... Plane parameter estimation part 44 ... Change amount calculation part 46 ... Conversion matrix calculation part 48 ... Difference image creation part 50 ... Moving target object detection part 52 ... Motion detectors 60a, 60b, 112 ... Roads 64a, 64b ... Pedestrians 70, 104 ... First reference position 72, 100 ... First region of interest 74, 106 ... Second reference position 76, 108 ... Second region of interest 116 ... Road pedestrian 118 ... Overpass pedestrian Ia ... first captured image Ib ... second captured image I (n-1), I (n), I (n + 1) ... captured image H ... planar projection matrix ↑ t ... translation Vector M ... human body R ... rotation matrix S ... road surface

Claims (10)

An imaging unit that is mounted on a vehicle and acquires a plurality of captured images by capturing images at different times while the vehicle is running;
Based on the position and orientation of the imaging unit when the first captured image is captured at the first imaging time point, the position and position of the imaging unit when the second captured image is captured at the second imaging time point A change amount calculation unit for calculating a translation vector and a rotation matrix representing the amount of change in posture;
A plane representing projective transformation from the first imaging surface indicated by the first captured image to the second imaging surface indicated by the second captured image based on the translation vector and the rotation matrix calculated by the change amount calculation unit. A transformation matrix calculation unit for calculating a projection matrix;
Vehicle motion component canceling means for canceling the motion component of the vehicle between the first captured image and the second captured image based on the planar projection matrix calculated by the conversion matrix calculating unit. A vehicle periphery monitoring device. The vehicle periphery monitoring device according to claim 1,
The transformation matrix calculation unit calculates the planar projection matrix representing projection transformation from a road area included in the first captured image to a road area included in the second captured image. Vehicle periphery monitoring device. In the vehicle periphery monitoring device according to claim 1 or 2,
The vehicle periphery monitoring device, wherein the vehicle motion component canceling unit is an image creating unit that creates an image in which the motion component of the vehicle is cancelled. In the vehicle periphery monitoring device according to claim 3,
A vehicle periphery monitoring device, further comprising a moving object detection unit that detects an object moving around the vehicle based on at least one of the images generated by the image generation unit. In the vehicle periphery monitoring device according to claim 4,
The said moving target object detection part detects the said target object based on the several said image from which at least one differs among the said 1st imaging time points and the said 2nd imaging time points, The vehicle periphery monitoring apparatus characterized by the above-mentioned. The vehicle periphery monitoring device according to any one of claims 3 to 5,
The vehicle periphery monitoring device, wherein the image creation unit creates a difference image as the image. In the vehicle periphery monitoring device according to any one of claims 3 to 6,
The image creation unit extracts a first region of interest including the object from the first captured image, extracts a second region of interest including the object from the second captured image, A vehicle periphery monitoring device that creates the image between a first region of interest and the second region of interest. In the vehicle periphery monitoring device according to claim 7,
The image creation unit specifies a position obtained by performing orthographic transformation based on the planar projection matrix with respect to a first reference position that specifies the position of the first region of interest, and specifies a position of the second region of interest. A vehicle periphery monitoring device, wherein the image is created as a reference position. In the vehicle periphery monitoring device according to claim 7,
The image creation unit specifies a position obtained by performing a reverse projection transformation based on the planar projection matrix with respect to a second reference position that specifies the position of the second region of interest, and specifies a position of the first region of interest. A vehicle periphery monitoring device, wherein the image is created as a reference position. In the vehicle periphery monitoring device according to any one of claims 3 to 9,
The image creation unit includes the captured image obtained by performing orthogonal projection transformation based on the planar projection matrix on the first captured image and the second captured image, and the second captured image. A vehicle periphery monitoring device that creates one of the images between a captured image subjected to reverse projection transformation based on a planar projection matrix and the first captured image.

Images