JP5272818B2 - Image processing apparatus and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve accuracy of detecting an object around a vehicle. <P>SOLUTION: Behavior estimation parts 212N to 212R of monitoring parts 201N to 201R estimate behavior parameters representing the behavior of a vehicle by using images obtained by imaging the different directions around the vehicle. A setting part 222 sets the behavior parameters to be used for detecting an obstacle around the vehicle on the basis the plurality of behavior parameters estimated by behavior estimation parts 212N to 212R. Obstacle detection parts 213N to 213R of the monitoring parts 201N to 201R detect the obstacle around the vehicle by using the images obtained by imaging the different directions around the vehicle and the set behavior parameters. This invention can be applied to, for example, an in-vehicle monitoring device. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to an image processing apparatus and method, and a program, and more particularly, to an image processing apparatus and method suitable for use in detecting an object around a vehicle, and a program.

  Conventionally, using an image taken by a camera attached to the front or rear of a vehicle such as an automobile, an obstacle existing in front or rear of the vehicle is detected, and a warning is given to the driver according to the detection result. A system has been developed.

  In such a system, the three-dimensional position of the object shown in the image is detected by the principle of triangulation using two images taken at different positions at different times by a single camera installed in the vehicle. A moving stereo system is often used (see, for example, Patent Document 1).

  Here, referring to FIG. 1, an image I1 taken at time t by a single camera installed in the vehicle and an image I2 taken at time t + 1 after a predetermined time Δt from time t. Consider a case where three-dimensional measurement is performed by a moving stereo method using two images.

  In the following, with respect to a coordinate system based on the camera (hereinafter referred to as a camera coordinate system), the horizontal direction of the image obtained as a result of shooting is the X axis, the vertical direction is the Y axis, and the direction perpendicular to the image plane is the Z axis And Also, with respect to a coordinate system based on the road surface (hereinafter referred to as a road surface coordinate system), the direction of translational movement of the vehicle is the Zg axis, the directions perpendicular to the Zg axis and perpendicular to the road surface are the Yg axis, the Zg axis, and the Yg axis. The vertical direction is taken as the Xg axis. Further, with respect to a coordinate system (hereinafter referred to as an image coordinate system) of an image captured by the camera, the horizontal direction is the x axis and the vertical direction is the y axis.

  When the vehicle moves between time t and time t + 1 and the optical center of the camera when the image I1 is captured and the optical center of the camera when the image I2 is captured are separated on the image I1 Based on the position P1 (xl, yl) of the object Q and the position P2 (xu, yu) of the object Q on the image I2, the principle of triangulation similar to the stereo camera system is used to The position of the dimension (X, Y, Z) can be detected. Further, the coordinate in the Z-axis direction obtained here is the distance from the camera (its optical center) to the object Q.

  By the way, in order to measure the three-dimensional position of an object around the vehicle by the moving stereo method, it is necessary to know the behavior of the camera while taking two images used for measurement. Here, the behavior of the camera means translational motion and rotational motion of the camera, and is almost equal to the translational motion and rotational motion of the vehicle in which the camera is installed.

  Assuming that the vehicle does not translate vertically and horizontally, the amount of translational motion in the longitudinal direction of the vehicle, that is, the amount of movement of the vehicle can be measured based on information from a vehicle speed sensor provided in the vehicle, for example. is there. And when the attachment position and attachment direction of the camera with respect to a vehicle are known, the translational momentum of the camera in a camera coordinate system can be calculated | required based on the moving amount | distance of a vehicle.

  However, since the vehicle speed sensor has low measurement accuracy, in order to increase the measurement accuracy, for example, the position of the road surface with respect to the camera is obtained using an image taken by the camera, and the vehicle is moved with reference to a point on the road surface. It is often done to estimate the quantity.

  The rotational momentum of the vehicle can be estimated using information from a vehicle speed sensor or a steering angle sensor provided in the vehicle, for example. And when the attachment position and attachment direction of the camera with respect to a vehicle are known, the rotational momentum of the camera in a camera coordinate system can be calculated | required based on the rotational momentum of a vehicle.

  However, since the vehicle speed sensor has a low measurement accuracy, in order to increase the measurement accuracy, for example, it is often performed to estimate the rotational momentum of the vehicle using an image taken by a camera. For example, the rotational momentum of the vehicle in the camera coordinate system (= the rotational momentum of the camera) is estimated using the following equation (1).

In Expression (1), x and y indicate the coordinates in the x-axis direction and the y-axis direction of the feature point of the object in the image coordinate system. v _x and v _y indicate components in the x-axis direction and the y-axis direction of the motion vector with respect to the feature point of the object in the image coordinate system. f indicates the focal length of the camera. t _x , t _y , and t _z indicate translational momentums in the X-axis, Y-axis, and Z-axis directions of the vehicle (camera) in the camera coordinate system. X, Y, and Z indicate the coordinates of the feature point of the object in the camera coordinate system in the X-axis direction, the Y-axis direction, and the Z-axis direction. θ, ψ, and φ are the rotational momentum of the vehicle (camera) in the camera coordinate system, that is, the rotation angle (pitch angle) around the X axis, the rotation angle (yaw angle) around the Y axis, and the rotation angle around the Z axis. (Roll angle) indicates φ.

JP 2000-74645 A

  However, when the translational momentum and the rotational momentum of the camera are estimated based on the image captured by the camera by the above-described method, the estimation accuracy may be deteriorated depending on how the image is reflected.

  For example, as shown in FIG. 2, when the wall approaches the shooting direction of the camera and the road surface hardly appears in the image, the estimation accuracy of the movement amount of the vehicle decreases, and accordingly, the estimation accuracy of the translational momentum of the camera. Decreases. Also, for example, as shown in FIG. 3, when the subject in the image is distorted due to the influence of the camera mounting position, the accuracy of estimation of the rotational momentum of the camera is lowered. As a result, the detection accuracy of objects around the vehicle decreases.

  The present invention has been made in view of such a situation, and makes it possible to improve detection accuracy of objects around a vehicle.

An image processing apparatus according to one aspect of the present invention is a behavior that represents a behavior of a vehicle in an image processing apparatus that detects an object around the vehicle using a plurality of images taken in different directions by a plurality of cameras provided in the vehicle. Based on at least one of behavior estimation means for estimating a parameter for each of a plurality of images, the traveling direction of the vehicle, the number of motion vectors used to estimate the behavior parameters, and the amount of the road surface in the images, A reliability evaluation means for evaluating the reliability of the parameters and a plurality of behavior parameters estimated for each of the plurality of images are weighted according to the respective reliability and averaged to be used for detecting objects around the vehicle. Setting means for setting behavior parameters, and object detection means for detecting objects around the vehicle using the behavior parameters set by the setting means and a plurality of images Including the.

In the image processing apparatus according to one aspect of the present invention, the behavior parameter indicating the behavior of the vehicle is estimated for each of a plurality of images taken in different directions, and the traveling direction of the vehicle and the number of motion vectors used for estimation of the behavior parameter are estimated. And the reliability of each behavior parameter is evaluated based on at least one of the road surface amounts in the image, and the plurality of behavior parameters estimated for each of the plurality of images are weighted according to each reliability. By averaging, behavior parameters used for detection of objects around the vehicle are set, and objects around the vehicle are detected using the set behavior parameters and a plurality of images.

Therefore, the detection accuracy of objects around the vehicle is improved. Further, it is possible to detect an object around the vehicle using a behavior parameter with higher accuracy, and as a result, detection accuracy is improved.

This behavior estimation means, setting means, reliability evaluation means, and object detection means are constituted by, for example, a CPU (Central Processing Unit).

  The image processing apparatus further includes road surface estimating means for estimating a road surface parameter representing the characteristics of the road surface on which the vehicle is traveling for each of a plurality of images, and the setting means includes a plurality of images estimated for each of the plurality of images. Based on the road surface parameter, the road surface parameter used for detection of objects around the vehicle is further set, and the object detection means uses the behavior parameter and road surface parameter set by the setting means, and a plurality of images. The object around the vehicle can be detected.

  Thereby, the detection accuracy of objects around the vehicle is further improved.

  This road surface estimation means is constituted by, for example, a CPU (Central Processing Unit).

The reliability evaluation unit, the traveling direction of the vehicle, the number of motion vectors used for the estimation of the behavior parameters and road parameters, and, based on at least one of the amount of the road surface in the image, each behavior parameters and road The parameter reliability is evaluated , and this setting means weights and averages a plurality of behavior parameters and road surface parameters in accordance with the respective reliability levels, so that the behavior parameters and road surface used for detection of objects around the vehicle are obtained. Parameters can be set.

  As a result, it is possible to detect an object around the vehicle using more accurate behavior parameters and road surface parameters, and as a result, detection accuracy is improved.

According to an image processing method of one aspect of the present invention, an image processing apparatus that detects an object around a vehicle using a plurality of images taken in different directions by a plurality of cameras provided in the vehicle is a behavior that represents the behavior of the vehicle. The parameter is estimated for each of a plurality of images , and the reliability of each behavior parameter is determined based on at least one of the traveling direction of the vehicle, the number of motion vectors used for estimating the behavior parameter, and the amount of the road surface in the image. The behavior parameters used for detection of objects around the vehicle are set and set by weighting and averaging the behavior parameters estimated for each of the images according to the respective reliability . Objects around the vehicle are detected using the behavior parameters and the plurality of images.

A program according to an aspect of the present invention provides a computer that detects an object around a vehicle using a plurality of images captured in different directions by a plurality of cameras provided on the vehicle, and sets a plurality of behavior parameters representing the behavior of the vehicle. Estimate for each image, evaluate the reliability of each behavior parameter based on at least one of the direction of travel of the vehicle, the number of motion vectors used to estimate the behavior parameter, and the amount of road surface in the image, A plurality of behavior parameters estimated for each of a plurality of images are weighted and averaged according to the respective reliability to set a behavior parameter used for detection of an object around the vehicle. A process including a step of detecting an object around the vehicle is executed using the image.

Therefore, the detection accuracy of objects around the vehicle is improved. Further, it is possible to detect an object around the vehicle using a behavior parameter with higher accuracy, and as a result, detection accuracy is improved.

This image processing method is executed by, for example, a CPU (Central Processing Unit).

  According to one aspect of the present invention, detection accuracy of objects around the vehicle is improved.

It is a figure for demonstrating the principle of the three-dimensional measurement by a moving stereo system. It is a figure which shows the example of the image image | photographed from the vehicle. It is a figure which shows the other example of the image image | photographed from the vehicle. It is a block diagram showing one embodiment of a monitoring system. It is a figure which shows the example of the installation position of a camera. It is a block diagram which shows 1st Embodiment of the image processing part of the monitoring system. It is a flowchart for demonstrating 1st Embodiment of the monitoring process. It is a block diagram which shows 2nd Embodiment of the image processing part of the monitoring system. It is a flowchart for demonstrating 2nd Embodiment of the monitoring process. It is a block diagram which shows 3rd Embodiment of the image processing part of the monitoring system. It is a flowchart for demonstrating 3rd Embodiment of the monitoring process. It is a block diagram which shows 4th Embodiment of the image processing part of the monitoring system. It is a flowchart for demonstrating 4th Embodiment of the monitoring process. It is a figure for demonstrating the correction process of a behavior parameter. It is a figure for demonstrating the correction process of a behavior parameter. And FIG. 11 is a block diagram illustrating an example of a configuration of a computer.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 4 is a block diagram showing an embodiment of an in-vehicle monitoring system to which the present invention is applied. FIG. 5 is a schematic view of the vehicle 131 provided with the monitoring system 101 of FIG. 4 as viewed from above.

  The monitoring system 101 in FIG. 4 is a system that monitors the surroundings of the vehicle 131, detects an obstacle existing around the vehicle 131, and outputs a detection result. The monitoring system 101 includes a camera 111N to a camera 111R, a gear detection unit 112, a steering angle detection unit 113, a vehicle speed detection unit 114, and an image processing unit 115.

  The cameras 111N to 111R are installed so as to photograph different directions around the vehicle 131, and supply images obtained as a result of the photographing to the image processing unit 115.

  Here, an example of installation positions of the cameras 111N to 111R will be described with reference to FIG. The camera 111N is installed, for example, at a position P1 that is approximately the center of the front end of the vehicle 131 (for example, approximately the center of the front bumper), and images the front of the vehicle 131. The camera 111CR is installed at a position P2 near the front end and the right end of the vehicle 131 (for example, near the right end of the front bumper), for example, and the vehicle 131 approaching from the right direction while stopping at the intersection can be seen. Direction). For example, the camera 111CL is installed at a position P3 in the vicinity of the front end and the left end of the vehicle 131 (for example, near the left end of the front bumper), so that a vehicle approaching from the left direction while stopping at the intersection can be seen. Direction). For example, the camera 111SR is installed at a position P4 on the right side surface of the vehicle 131 (for example, near the right door mirror), and captures the right direction of the vehicle 131. The camera 111SL is installed, for example, at a position P5 on the left side surface of the vehicle 131 (for example, near the left door mirror), and photographs the left direction of the vehicle 131. For example, the camera 111 </ b> R is installed at a position P <b> 6 that is approximately the center of the rear end of the vehicle 131 (for example, approximately the center of the rear bumper) and captures the rear of the vehicle 131.

  Hereinafter, an image captured by the camera 111N is referred to as a front image, an image captured by the camera 111CR is referred to as a front right image, an image captured by the camera 111CL is referred to as a front left image, and is captured by the camera 111SR. An image captured by the camera 111SL is referred to as a left image, and an image captured by the camera 111R is referred to as a rear image. Hereinafter, when it is not necessary to individually distinguish the cameras 111N to 111R, they are simply referred to as the camera 111.

  The gear detection unit 112 detects the gear position (or shift position) of the vehicle 131 and notifies the image processing unit 115 of the detection result.

  The steering angle detection unit 113 includes, for example, a steering angle sensor, detects the steering angle of the vehicle 131, and notifies the image processing unit 115 of the detection result.

  The vehicle speed detection unit 114 includes, for example, a vehicle speed sensor, detects the speed of the vehicle 131, and notifies the image processing unit 115 of the detection result.

  The image processing unit 115 detects obstacles around the vehicle 131 based on the images taken by the cameras 111 and the position, steering angle, and speed of the gear of the vehicle 131. Further, the image processing unit 115 outputs the detection result to a subsequent apparatus that uses the detection result of the obstacle.

  FIG. 6 is a block diagram illustrating a functional configuration of the image processing unit 115 </ b> A that is the first embodiment of the image processing unit 115. The image processing unit 115A is configured to include monitoring units 201N to 201R and a monitoring control unit 202. The monitoring unit 201N includes a road surface estimation unit 211N, a behavior estimation unit 212N, and an obstacle detection unit 213N. The monitoring unit 201CR includes a road surface estimation unit 211CR, a behavior estimation unit 212CR, and an obstacle detection unit 213CR. 201CL includes a road surface estimation unit 211CL, a behavior estimation unit 212CL, and an obstacle detection unit 213CL. The monitoring unit 201SR includes a road surface estimation unit 211SR, a behavior estimation unit 212SR, and an obstacle detection unit 213SR. The monitoring unit 201SL includes a road surface. The monitoring unit 201R includes an estimation unit 211SL, a behavior estimation unit 212SL, and an obstacle detection unit 213SL, and the monitoring unit 201R is configured to include a road surface estimation unit 211R, a behavior estimation unit 212R, and an obstacle detection unit 213R. Furthermore, the monitoring control unit 202 is configured to include a traveling direction detection unit 221 and a setting unit 222.

  The monitoring unit 201N detects an obstacle ahead of the vehicle 131 based on the front image supplied from the camera 111N and the speed of the vehicle 131 notified from the vehicle speed detection unit 114.

  More specifically, the road surface estimation unit 211N estimates various parameters (hereinafter referred to as road surface parameters) representing the characteristics of the road surface ahead of the vehicle 131 based on the front image and the speed of the vehicle 131.

  The behavior estimation unit 212N estimates various parameters (hereinafter referred to as behavior parameters) representing the behavior of the vehicle 131 (hereinafter also referred to as own vehicle behavior) based on the front image.

  Then, the monitoring unit 201N transmits estimation information including the estimated road surface parameter and behavior parameter to the monitoring control unit 202.

  The obstacle detection unit 213N detects an obstacle ahead of the vehicle 131 based on the estimated information transmitted from the monitoring control unit 202 and the front image. The obstacle detection unit 213N outputs the detection result to a subsequent apparatus.

  The monitoring unit 201CR, the monitoring unit 201CL, the monitoring unit 201SR, the monitoring unit 201SL, and the monitoring unit 201R have the same configuration and function as the monitoring unit 201N, and detailed description thereof will be omitted.

  Note that the monitoring unit 201CR detects an obstacle in the right front direction of the vehicle 131 using the front right image supplied from the camera 111CR. The monitoring unit 201CL detects an obstacle in the left front direction of the vehicle 131 using the front left image supplied from the camera 111CL. Monitoring unit 201SR detects an obstacle in the right direction of vehicle 131 using the right image supplied from camera 111SR. The monitoring unit 201SL detects an obstacle in the left direction of the vehicle 131 using the left image supplied from the camera 111SL. The monitoring unit 201R detects an obstacle behind the vehicle 131 using the rear image supplied from the camera 111R.

  Hereinafter, the monitoring units 201N to 201R are simply referred to as the monitoring unit 201 when it is not necessary to distinguish them individually. Hereinafter, when it is not necessary to individually distinguish the road surface estimation units 211N to 211R, they are simply referred to as a road surface estimation unit 211. Furthermore, hereinafter, when it is not necessary to individually distinguish the behavior estimation units 212N to 212R, they are simply referred to as the behavior estimation unit 212. In the following description, the obstacle detection units 213N to 213R are simply referred to as the obstacle detection unit 213 when it is not necessary to distinguish them individually.

  The monitoring control unit 202 controls the operation of each monitoring unit 201 while giving various commands and information to each monitoring unit 201 and acquiring various types of information from each monitoring unit 201.

  More specifically, the traveling direction detection unit 221 of the monitoring control unit 202 acquires information indicating the gear position of the vehicle 131 from the gear detection unit 112, and information indicating the steering angle of the vehicle 131 from the steering angle detection unit 113. To get. The traveling direction detection unit 221 detects the traveling direction of the vehicle 131 based on the gear position of the vehicle 131 and the steering angle of the vehicle 131 and notifies the setting unit 222 of the detection result.

  The setting unit 222 sets estimation information used by each obstacle detection unit 213 for detecting an obstacle based on the traveling direction of the vehicle 131. More specifically, the setting unit 222 selects the monitoring unit 201 that estimates the own vehicle behavior and the road surface based on the traveling direction of the vehicle 131, and estimates the own vehicle behavior and the road surface to the selected monitoring unit 201. Command. Then, the setting unit 222 acquires estimation information from the commanded monitoring unit 201, transmits the estimated information to each monitoring unit 201, and commands each monitoring unit 201 to detect an obstacle.

  Next, the monitoring process executed by the monitoring system 101 will be described with reference to the flowchart of FIG. This process is started when, for example, the accessory (ACC) of the vehicle 131 is turned on, and is ended when the accessory of the vehicle 131 is turned off.

  In step S <b> 1, the traveling direction detection unit 221 detects the traveling direction of the vehicle 131. Specifically, first, the traveling direction detection unit 221 acquires information indicating the gear position of the vehicle 131 from the gear detection unit 112, and acquires information indicating the steering angle of the vehicle 131 from the steering angle detection unit 113.

  For example, the traveling direction detector 221 is configured such that when the gear of the vehicle 131 is set to a forward gear and the steering angle is within a predetermined range (for example, within 10 ° to the left and right), the vehicle 131 goes straight ahead. It is determined that it is in the middle. Further, the traveling direction detection unit 221 is configured such that when the gear of the vehicle 131 is set to a reverse gear and the steering angle is within a predetermined range (for example, within 10 ° to the left and right), the vehicle 131 goes straight backward. It is determined that it is in the middle. Further, the traveling direction detection unit 221 is turning the vehicle 131 to the right when the gear of the vehicle 131 is set to a forward or backward gear and the steering angle exceeds a predetermined range in the right direction. Is determined. The traveling direction detection unit 221 is turning the vehicle 131 to the left when the gear of the vehicle 131 is set to a forward or backward gear and the steering angle exceeds a predetermined range in the left direction. Is determined. Further, the traveling direction detection unit 221 determines that the vehicle 131 is stopped when the gear of the vehicle 131 is not set to either the forward gear or the reverse gear.

  Then, the traveling direction detection unit 221 notifies the setting unit 222 of the detection result of the traveling direction of the vehicle 131.

  In step S <b> 2, the setting unit 222 determines whether the vehicle 131 is traveling straight ahead based on the detection result by the traveling direction detection unit 221. If it is determined that the vehicle 131 is traveling straight ahead, the process proceeds to step S3.

  In step S3, the image processing unit 115A acquires an image. In other words, the setting unit 222 instructs each monitoring unit 201 to acquire an image, and each monitoring unit 201 acquires an image from the corresponding camera 111 at the same timing in synchronization according to the command. .

  In step S <b> 4, the monitoring unit 201 </ b> R estimates a road surface area using an image obtained by photographing the rear of the vehicle 131. Specifically, the setting unit 222 instructs the monitoring unit 201R to estimate the vehicle behavior and the road surface. The road surface estimation unit 211R of the monitoring unit 201R acquires information indicating the speed of the vehicle 131 from the vehicle speed detection unit 114. The road surface estimation unit 211R estimates the amount of movement of the vehicle 131 since the previous frame was shot based on the speed of the vehicle 131, and using the result, the rear image acquired from the camera 111R by a predetermined method. The area in which the road surface is reflected is estimated.

  Note that the method of estimating the road surface area is not limited to a specific method, and any method can be adopted.

  In step S <b> 5, the monitoring unit 201 </ b> R estimates the behavior of the vehicle 131 and the road surface using an image obtained by photographing the rear of the vehicle 131. Specifically, the road surface estimation unit 211R of the monitoring unit 201R estimates road surface parameters representing road surface characteristics for the road surface region estimated in step S4 by a predetermined method. The road surface parameters are the distance between the camera 111R and the road surface (for example, the distance between the road surface when the road surface is regarded as a plane and the optical center of the camera 111), and the camera 111R and the road surface such as the inclination of the road surface. Information indicating the positional relationship between the road surface, the shape of the road surface, the amount (area) of the road surface in the image, the state of the road surface, and the like.

  In addition, the behavior estimation unit 212R of the monitoring unit 201R estimates the behavior parameters of the vehicle 131. For example, the behavior estimation unit 212R extracts a feature point of the rear image one frame before (for example, an edge or a corner shown in the rear image), and corresponds to the extracted feature point in the rear image of the current frame. Search for a point. In addition, the behavior estimation unit 212R calculates a motion vector that connects the extracted feature point and the searched corresponding point. Then, the behavior estimation unit 212R uses the road surface parameter estimated by the road surface estimation unit 211R as necessary, for example, using the above-described equation (1), the pitch angle θ representing the rotational momentum of the vehicle 131, yaw The behavior parameters including the angle ψ and the roll angle φ are estimated.

  Then, the monitoring unit 201R transmits estimation information including the estimated behavior parameter and road surface parameter to the monitoring control unit 202.

  In addition, the estimation method of a road surface parameter and a behavior parameter is not limited to a specific method, It is possible to employ | adopt arbitrary methods.

  Here, the reason why the vehicle behavior and the road surface are estimated using a rear image obtained by photographing the rear of the vehicle 131 when the vehicle 131 is traveling straight ahead is described. When the vehicle 131 is traveling on a road, the road surface appears almost certainly after the vehicle 131 passes in the opposite direction, that is, after the vehicle 131 passes. Therefore, when the vehicle 131 is traveling straight ahead, it is estimated that the road surface is almost certainly reflected in the rear image and that the amount (area) is large. Therefore, by using a rear image that is estimated to have a large amount of road surface in the image, it can be expected that the estimation accuracy of the road surface parameter representing the characteristics of the road surface on which the vehicle 131 is traveling is improved.

  In addition, the road surface is easy to grasp the positional relationship with the vehicle 131, and in addition, there is a high possibility that an object having many feature points exists on or around the road surface. Therefore, it can be expected that the estimation accuracy of the behavior parameter representing the behavior of the vehicle 131 is improved by using the rear image estimated to have a large amount of road surface in the image.

  For the above reasons, when the vehicle 131 is traveling straight ahead, the vehicle behavior and the road surface are estimated using the rear image.

  In step S <b> 6, the setting unit 222 transmits estimation information to each monitoring unit 201. That is, the setting unit 222 transmits the estimation information received from the monitoring unit 201R to each monitoring unit 201 and commands the detection of an obstacle. At this time, the estimation information may not be transmitted to the monitoring unit 201R that is the transmission source of the estimation information. Thereafter, the process proceeds to step S22.

  On the other hand, if it is determined in step S2 that the vehicle 131 is not traveling straight ahead, the process proceeds to step S7.

  In step S <b> 7, the setting unit 222 determines whether or not the vehicle 131 is traveling straight backward based on the detection result by the traveling direction detection unit 221. When it is determined that the vehicle 131 is traveling straight backward, the process proceeds to step S8.

  In step S <b> 8, each monitoring unit 201 acquires an image from each camera 111 as in the process of step S <b> 3.

  In step S9, the monitoring unit 201N performs the same processing as the monitoring unit 201R in step S4 based on the instruction of the setting unit 222, and uses the front image obtained by photographing the front of the vehicle 131 with the camera 111N to determine the road surface area. presume.

  In step S10, the monitoring unit 201N performs the same processing as that of the monitoring unit 201R in step S5, and estimates the behavior of the vehicle 131 and the road surface using the front image. Then, the monitoring unit 201N transmits estimation information including the estimated behavior parameter and road surface parameter to the monitoring control unit 202.

  For the same reason as the case where the vehicle 131 is traveling straight ahead, when the vehicle 131 is traveling straight behind, it is estimated that the road surface is almost certainly reflected in the front image and the amount thereof is large. Therefore, the vehicle behavior and the road surface are estimated using the front image.

  In step S11, processing similar to that in step S6 is performed, and estimated information estimated by the monitoring unit 201N is transmitted to each monitoring unit 201. Thereafter, the process proceeds to step S22.

  On the other hand, when it is determined in step S7 that the vehicle 131 is not traveling straight backward, the process proceeds to step S12.

  In step S12, the setting unit 222 determines whether or not the vehicle 131 is turning right based on the detection result by the traveling direction detection unit 221. If it is determined that the vehicle 131 is turning right, the process proceeds to step S13.

  In step S <b> 13, each monitoring unit 201 acquires an image from each camera 111 as in the process of step S <b> 3.

  In step S14, the monitoring unit 201CL performs a process similar to that of the monitoring unit 201R in step S4 based on a command from the setting unit 222, and uses the left front image of the vehicle 131 photographed by the camera 111CL to determine the road surface area. presume.

  In step S15, the monitoring unit 201CL performs the same processing as the monitoring unit 201R in step S5, and estimates the behavior of the vehicle 131 and the road surface using the front left image. Then, the monitoring unit 201CL transmits estimation information including the estimated behavior parameter and road surface parameter to the monitoring control unit 202.

  For the same reason as the case where the vehicle 131 is traveling straight ahead, when the vehicle 131 is turning right, it is estimated that the road surface is almost certainly reflected in the front left image and the amount thereof is large. Therefore, the vehicle behavior and the road surface are estimated using the left front image.

  In step S16, processing similar to that in step S6 is performed, and estimated information estimated by the monitoring unit 201CL is transmitted to each monitoring unit 201. Thereafter, the process proceeds to step S22.

  On the other hand, if it is determined in step S12 that the vehicle 131 is not turning right, the process proceeds to step S17.

  In step S <b> 17, the setting unit 222 determines whether or not the vehicle 131 is turning left based on the detection result by the traveling direction detection unit 221. If it is determined that the vehicle 131 is turning left, the process proceeds to step S18.

  In step S <b> 18, each monitoring unit 201 acquires an image from each camera 111 as in the process of step S <b> 3.

  In step S19, the monitoring unit 201CR performs the same processing as the monitoring unit 201R in step S4 based on the instruction of the setting unit 222, and uses the front right image obtained by photographing the front right direction of the vehicle 131 with the camera 111CR. Estimate the road surface area.

  In step S20, the monitoring unit 201CR performs the same processing as the monitoring unit 201R in step S5, and estimates the behavior of the vehicle 131 and the road surface using the front right image. The monitoring unit 201CR transmits estimation information including the estimated behavior parameter and road surface parameter to the monitoring control unit 202.

  For the same reason as the case where the vehicle 131 is traveling straight ahead, when the vehicle 131 is turning to the left, it is estimated that the road surface is almost certainly reflected in the front right image and the amount thereof is large. Therefore, the vehicle behavior and the road surface are estimated using the front right image.

  In step S21, processing similar to that in step S6 is performed, and estimated information estimated by the monitoring unit 201CR is transmitted to each monitoring unit 201. Thereafter, the process proceeds to step S22.

  In step S <b> 22, the image processing unit 115 </ b> A detects an obstacle in each monitoring unit 201 using the transmitted estimation information. For example, the obstacle detection unit 213N of the monitoring unit 201N exists in front of the vehicle 131 using the behavior parameter and the road surface parameter included in the estimation information received from the setting unit 222 and the front image captured by the camera 111N. And detecting three-dimensional information of the detected obstacle (that is, the position of the obstacle in the road surface coordinate system). Similarly, the obstacle detection unit 213 of the other monitoring unit 201 also captures an image in which the behavior parameter and the road surface parameter included in the estimation information received from the setting unit 222 and the direction to be monitored by each monitoring unit 201 are captured. The obstacles present in the direction to be monitored by each monitoring unit 201 and the three-dimensional information of the detected obstacles are detected.

  The method for detecting the obstacle is not limited to a specific method, and any method can be adopted. Moreover, you may make it detect the attribute of an obstruction other than the three-dimensional information of an obstruction.

  Thereafter, the process returns to step S1, and the processes of steps S1 to S22 are repeatedly executed.

  On the other hand, if it is determined in step S17 that the vehicle 131 is not turning left, that is, if it is determined that the vehicle is stopped, no processing is performed, and the processing returns to step S1. The processes after S1 are executed.

  As described above, the estimation accuracy can be improved by estimating the behavior parameter and the road surface parameter using the image obtained by photographing the direction closer to the direction opposite to the traveling direction of the vehicle. As a result, obstacles in each direction around the vehicle 131 can be detected using highly accurate behavior parameters and road surface parameters, and the accuracy of detecting obstacles in each direction around the vehicle 131 is improved.

  In addition, since each monitoring unit 201 acquires an image at the same timing synchronously and detects an obstacle using the image, the estimation estimated using the images acquired at the same time by other monitoring units 201 Obstacles can be detected using the information. As a result, it is possible to suppress a decrease in detection accuracy caused by using the estimation information of the other monitoring unit 201.

  Next, a second embodiment of the present invention will be described with reference to FIGS.

  FIG. 8 is a block diagram illustrating a functional configuration of an image processing unit 115B that is the second embodiment of the image processing unit 115. The image processing unit 115B is configured to include monitoring units 201N to 201R and a monitoring control unit 301. In addition, the monitoring control unit 301 is configured to include a setting unit 311. In the figure, parts corresponding to those in FIG. 6 are denoted by the same reference numerals, and description of parts having the same processing will be omitted because it will be repeated.

  The image processing unit 115B is different from the image processing unit 115A in FIG. 6 in that a monitoring control unit 301 is provided instead of the monitoring control unit 202.

  The monitoring control unit 301 controls the operation of each monitoring unit 201 while giving various commands and information to each monitoring unit 201 and acquiring various types of information from each monitoring unit 201.

  More specifically, the setting unit 311 of the monitoring control unit 301 instructs each monitoring unit 201 to estimate the own vehicle behavior and the road surface, and displays information indicating the number of motion vectors used for the own vehicle behavior and the road surface estimation. Obtained from each monitoring unit 201. Then, the setting unit 311 selects estimation information to be used based on the own vehicle behavior and the number of motion vectors used for estimating the road surface, and acquires the selected estimation information from the monitoring unit 201 that estimated the estimation information. Then, the information is transmitted to each monitoring unit 201 and the monitoring unit 201 is commanded to detect an obstacle.

  Next, the monitoring process executed by the monitoring system 101 including the image processing unit 115B will be described with reference to the flowchart of FIG. This process is started when, for example, the accessory (ACC) of the vehicle 131 is turned on, and is ended when the accessory of the vehicle 131 is turned off.

  In step S <b> 51, each monitoring unit 201 of the image processing unit 115 </ b> B acquires an image from each camera 111, similarly to the processing in step S <b> 3 in FIG. 7.

  In step S <b> 52, the image processing unit 115 </ b> B uses the respective monitoring units 201 to estimate the own vehicle behavior and the road surface. Specifically, the setting unit 311 instructs each monitoring unit 201 to estimate the own vehicle behavior and the road surface. The road surface estimation unit 211 of each monitoring unit 201 performs processing similar to that of the monitoring unit 201R in steps S4 and S5 in FIG. 7 to estimate road surface parameters. At this time, the estimation process of the road surface area may be omitted. In addition, the behavior estimation unit 212 of each monitoring unit 201 performs the same processing as the monitoring unit 201R in step S5 of FIG. 7, and estimates behavior parameters.

  In step S <b> 53, the setting unit 311 acquires from each monitoring unit 201 the number of motion vectors used by each monitoring unit 201 to estimate the vehicle behavior and the road surface. In other words, the setting unit 311 acquires from each monitoring unit 201 the number of motion vectors that were effective in estimating the vehicle behavior and the road surface.

  In step S <b> 54, the setting unit 311 transmits the estimation information of the monitoring unit 201 having the largest number of used motion vectors to each monitoring unit 201. Specifically, the setting unit 311 identifies the monitoring unit 201 having the largest number of motion vectors used for estimating the vehicle behavior and the road surface based on the information acquired in the process of step S53, and identifies the identified monitoring. The estimation information is acquired from the unit 201. Then, the setting unit 311 transmits the acquired estimated information to each monitoring unit 201 and commands the detection of an obstacle. At this time, the estimation information may not be transmitted from the setting unit 311 to the monitoring unit 201 that is a transmission source of the estimation information.

  Here, as the number of used motion vectors increases, the vehicle behavior and the road surface are estimated based on a larger amount of information, and it is considered that the estimation accuracy is higher. Therefore, the estimation information of the monitoring unit 201 that has used the largest number of motion vectors is selected as the estimation information used for detecting the obstacle.

  In step S55, similarly to the process in step S22 of FIG. 7, each monitoring unit 201 detects an obstacle using the transmitted estimated information. That is, each monitoring unit 201 detects an obstacle using the behavior parameter and the road surface parameter estimated by the monitoring unit 201 having the largest number of motion vectors used for estimating the vehicle behavior and the road surface.

  Thereafter, the process returns to step S51, and the processes of steps S51 to S55 are repeatedly executed.

  In this way, obstacles in each direction around the vehicle 131 can be detected using more accurate behavior parameters and road surface parameters, and the obstacle detection accuracy in each direction around the vehicle 131 can be detected. improves.

  Next, a third embodiment of the monitoring system 101 will be described with reference to FIGS. 10 and 11.

  FIG. 10 is a block diagram illustrating a functional configuration of an image processing unit 115 </ b> C that is the third embodiment of the image processing unit 115. The image processing unit 115C is configured to include monitoring units 201N to 201R and a monitoring control unit 401. In addition, the monitoring control unit 401 is configured to include a setting unit 411. In the figure, parts corresponding to those in FIG. 6 are denoted by the same reference numerals, and description of parts having the same processing will be omitted because it will be repeated.

  The image processing unit 115C is different from the image processing unit 115A in FIG. 6 in that a monitoring control unit 401 is provided instead of the monitoring control unit 202.

  The monitoring control unit 401 controls the operation of each monitoring unit 201 while giving various commands and information to each monitoring unit 201 and acquiring various types of information from each monitoring unit 201.

  More specifically, the setting unit 411 of the monitoring control unit 401 instructs each monitoring unit 201 to estimate the own vehicle behavior and the road surface, and to detect an obstacle. The setting unit 411 acquires information indicating the amount of road surface detected from the image by the obstacle detection process from each monitoring unit 201. The setting unit 411 selects the monitoring unit 201 that estimates the vehicle behavior and the road surface based on the detected amount of the road surface, and instructs the selected monitoring unit 201 to estimate the vehicle behavior and the road surface. Then, the setting unit 411 acquires the estimation information from the commanded monitoring unit 201 and transmits the estimation information to each monitoring unit 201, and commands each monitoring unit 201 to detect an obstacle.

  Next, monitoring processing executed by the monitoring system 101 including the image processing unit 115C will be described with reference to the flowchart of FIG. This process is started when, for example, the accessory (ACC) of the vehicle 131 is turned on, and is ended when the accessory of the vehicle 131 is turned off.

  In step S <b> 101, each monitoring unit 201 of the image processing unit 115 </ b> C acquires an image from each camera 111, similarly to the processing in step S <b> 3 in FIG. 7.

  In step S102, the own vehicle behavior and the road surface are estimated in each monitoring unit 201 as in the process of step S52 of FIG.

  In step S <b> 103, the image processing unit 115 </ b> C detects an obstacle in each monitoring unit 201 using the estimation information obtained in each monitoring unit 201. That is, each monitoring unit 201 of the image processing unit 115C detects an obstacle by the same process as step S22 in FIG. 7 using the own vehicle behavior and road surface estimation information obtained by itself.

  In step S104, the setting unit 411 acquires the detected amount of road surface. That is, the setting unit 411 acquires information indicating the amount of road surface detected when each obstacle detection unit 213 detects an obstacle from each monitoring unit 201.

  In step S <b> 105, each monitoring unit 201 of the image processing unit 115 </ b> C acquires an image from each camera 111 as in the process of step S <b> 3 in FIG. 7.

  In step S <b> 106, the image processing unit 115 </ b> C estimates the vehicle behavior and the road surface with the monitoring unit 201 that has detected the largest amount of road surface. Specifically, the setting unit 411 identifies the monitoring unit 201 in which the amount of road surface detected by each obstacle detection unit 213 is maximum based on the information acquired in the process of step S104. The setting unit 411 instructs the identified monitoring unit 201 to estimate the own vehicle behavior and the road surface, and obtains estimation information indicating the result. Then, the setting unit 411 transmits the acquired estimation information to each monitoring unit 201 and commands the detection of an obstacle. At this time, the estimation information may not be transmitted to the monitoring unit 201 that is the transmission source of the estimation information.

  Here, the monitoring unit 201 that has detected the largest amount of road surface next uses the next frame, that is, the image of the next frame in which the direction to be monitored by the monitoring unit 201 is captured. It is expected to be reflected. Further, as described above, the larger the amount of road surface in the image, the higher the estimation accuracy of behavior parameters and road surface parameters can be expected. Therefore, the estimated information of the monitoring unit 201 that has detected the maximum amount of road surface is selected as estimated information used for detecting an obstacle.

  In step S <b> 107, the setting unit 411 transmits the obtained estimation information to each monitoring unit 201. That is, the setting unit 411 transmits the estimated information acquired from the monitoring unit 201 having the largest amount of road surface detected at the time of the previous obstacle detection to the monitoring unit 201 in the process of step S106, and Command detection. At this time, the estimation information may not be transmitted to the monitoring unit 201 that is the transmission source of the estimation information.

  In step S108, similarly to the process in step S22 of FIG. 7, each monitoring unit 201 detects an obstacle using the transmitted estimated information. That is, each monitoring unit 201 detects an obstacle using the behavior parameter and the road surface parameter estimated by the monitoring unit 201 in which the amount of the road surface detected at the time of the previous obstacle detection is the maximum.

  Thereafter, the process returns to step S104, and the processes of steps S104 to S108 are repeatedly executed.

  In this way, obstacles in each direction around the vehicle 131 can be detected using more accurate behavior parameters and road surface parameters, and the obstacle detection accuracy in each direction around the vehicle 131 can be detected. improves.

  Next, a fourth embodiment of the monitoring system 101 will be described with reference to FIGS.

  FIG. 12 is a block diagram illustrating a functional configuration of an image processing unit 115D which is the fourth embodiment of the image processing unit 115. As illustrated in FIG. The image processing unit 115D is configured to include monitoring units 201N to 201R and a monitoring control unit 501. The monitoring control unit 501 is configured to include a traveling direction detection unit 511, a reliability evaluation unit 512, and a setting unit 513. In the figure, parts corresponding to those in FIG. 6 are denoted by the same reference numerals, and description of parts having the same processing will be omitted because it will be repeated.

  The image processing unit 115D is different from the image processing unit 115A in FIG. 6 in that a monitoring control unit 501 is provided instead of the monitoring control unit 202.

  The monitoring control unit 501 controls the operation of each monitoring unit 201 while giving various commands and information to each monitoring unit 201 and acquiring various types of information from each monitoring unit 201.

  More specifically, the traveling direction detection unit 511 of the monitoring control unit 501 has the same function as the traveling direction detection unit 221 of the monitoring control unit 202 of FIG. Based on the corner, the traveling direction of the vehicle 131 is detected and the detection result is notified to the reliability evaluation unit 512.

  The reliability evaluation unit 512 acquires from each monitoring unit 201 the number of motion vectors used by each monitoring unit 201 to estimate the vehicle behavior and the road surface. Further, the reliability evaluation unit 512 acquires the estimation information acquired by the setting unit 513 from each monitoring unit 201 from the setting unit 513. The reliability evaluation unit 512 is estimated by the traveling direction of the vehicle 131, the number of motion vectors used by each monitoring unit 201 for estimating the vehicle behavior and the road surface, and the road surface estimation unit 211 of each monitoring unit 201. Based on the road surface amount, the reliability of the estimation information of each monitoring unit 201 is evaluated. The reliability evaluation unit 512 notifies the setting unit 513 of the reliability of the estimated information of each monitoring unit 201 that has been evaluated.

  The setting unit 513 instructs each monitoring unit 201 to estimate the vehicle behavior and the road surface, and obtains estimation information indicating the result from each monitoring unit 201. Then, the setting unit 513 sets estimated information used by each monitoring unit 201 for detecting an obstacle based on the reliability of the estimated information of each monitoring unit 201 evaluated by the reliability evaluation unit 512. The setting unit 311 transmits the set estimation information to each monitoring unit 201 and instructs each monitoring unit 201 to detect an obstacle.

  Next, the monitoring process executed by the monitoring system 101 including the image processing unit 115D will be described with reference to the flowchart of FIG. This process is started when, for example, the accessory (ACC) of the vehicle 131 is turned on, and is ended when the accessory of the vehicle 131 is turned off.

  In step S151, each monitoring unit 201 of the image processing unit 115D acquires an image from each camera 111, similarly to the processing in step S3 in FIG.

  In step S152, as in the process of step S52 of FIG. 9, each monitoring unit 201 estimates the vehicle behavior and the road surface, and the estimated information obtained as a result is transmitted from each monitoring unit 201 to the monitoring control unit 501. Is done.

  In step S153, the reliability evaluation unit 512 evaluates the reliability of the estimation information of each monitoring unit 201. Specifically, the setting unit 513 supplies the estimation information acquired from each monitoring unit 201 to the reliability evaluation unit 512. The traveling direction detection unit 221 performs the same processing as the traveling direction detection unit 221 in step S1 of FIG. 7, detects the traveling direction of the vehicle 131, and notifies the reliability evaluation unit 512 of the detected traveling direction. In addition, the reliability evaluation unit 512 performs the same processing as the setting unit 311 in step S <b> 53 of FIG. 9, and acquires the number of motion vectors used for estimating the vehicle behavior and the road surface from each monitoring unit 201.

  Then, the reliability evaluation unit 512 includes the traveling direction of the vehicle 131, the number of motion vectors used by each monitoring unit 201 to estimate the vehicle behavior and the road surface, and each road surface included in the estimation information from each monitoring unit 201. Based on the road surface amount estimated by the estimation unit 211, the reliability of the estimation information of each monitoring unit 201 is evaluated.

  At this time, as described above, it is considered that the estimation accuracy is higher as the shooting direction of the image used for estimating the vehicle behavior and the road surface is closer to the direction opposite to the traveling direction of the vehicle 131, and thus the reliability is high. Be evaluated. On the contrary, it is evaluated that the reliability is lower as the shooting direction of the image used for estimating the vehicle behavior and the road surface is closer to the traveling direction of the vehicle.

  In addition, the greater the number of motion vectors used for estimating the vehicle behavior and the road surface, the higher the reliability is evaluated. The smaller the number of motion vectors used for estimating the vehicle behavior and the road surface, the lower the reliability. It is evaluated.

  Furthermore, as described above, since the estimation accuracy is considered to be higher as the road surface amount in the image used for estimation of the vehicle behavior and the road surface is larger, the reliability is higher as the detected road surface amount is larger. The smaller the detected and detected road surface amount, the lower the reliability.

  The reliability evaluation unit 512 notifies the setting unit 513 of the reliability of the estimated information of each monitoring unit 201 evaluated in this way.

  In addition, without using all of the traveling direction of the vehicle 131, the number of motion vectors used for estimating the vehicle behavior and the road surface, and the detected road surface amount, using one or two of them, The reliability of the estimation information may be evaluated. Moreover, you may make it evaluate reliability using other information. Furthermore, the reliability may be individually evaluated based on different evaluation criteria for the behavior parameter and the road surface parameter. Further, the reliability may be individually evaluated for each parameter based on different evaluation criteria.

  In step S154, the setting unit 513 sets the estimation information based on the reliability of the estimation information of each monitoring unit 201. Specifically, the setting unit 513 averages the behavior parameters estimated by the respective monitoring units 201 in accordance with the respective reliability, and the obtained behavior parameters are obstructed by the respective monitoring units 201. Set to the behavior parameter used for detection. Further, the setting unit 513 averages the road surface parameters estimated by the respective monitoring units 201 in accordance with the respective reliability, and the obtained road surface parameters are detected by the respective monitoring units 201 for obstacle detection. Set to the road surface parameter to be used.

  In step S155, the setting unit 513 transmits estimation information including the set behavior parameters and road surface parameters to each monitoring unit 201.

  In step S156, similarly to the process of step S22 of FIG. 7, each monitoring unit 201 detects an obstacle using the transmitted estimated information.

  Thereafter, the process returns to step S151, and the processes of steps S151 to S156 are repeatedly executed.

  In this way, obstacles in each direction around the vehicle 131 can be detected using more accurate behavior parameters and road surface parameters, and the obstacle detection accuracy in each direction around the vehicle 131 can be detected. improves.

  By the way, each camera 111 may move differently depending on the mounting position on the vehicle 131. This point will be described with reference to FIG.

  FIG. 14 is a schematic view of the vehicle 601 as viewed from above, and the positions of the front wheels 611L and 611R and the rear wheels 612L and 612R are shown superimposed on the vehicle 601 in order to make the positions of the wheels easy to understand. .

  For example, when the vehicle 601 starts to turn leftward, the camera B moves in an obliquely leftward direction as indicated by an arrow, while the camera A substantially forwards as indicated by an arrow. There are times when you go straight on. As described above, when the behavior of the camera A and the behavior of the camera B are different, the monitoring unit 201 that detects an obstacle using the image captured by the camera B is estimated using the image captured by the camera A. When using the behavior parameter, it is desirable to correct the behavior parameter for use.

  Here, an example of a behavior parameter correction method will be described with reference to FIG. In the following description, an axis having a positive direction as the direction from the rear to the front of the vehicle 601 is defined as a Z axis, and an axis perpendicular to the Z axis and having a direction from the left to the right of the vehicle 601 as a positive direction is an X axis. An axis that is perpendicular to the X axis and the Z axis and has a positive direction from the bottom to the top of the vehicle 601 is defined as a Y axis.

Hereinafter, the angle of the front wheels 611L and 611R with respect to the X-axis direction is α. The angle α can be detected by the steering angle detector 113, for example. Further, the wheel base length of the vehicle 601 that is the distance between the center of the front wheel 611R and the center of the rear wheel 612R (or the distance between the center of the front wheel 611L and the center of the rear wheel 612L) is defined as B. Further, the difference in the X-axis direction between the mounting position of the camera A and the position of the rear wheel 612R of the vehicle 601 is set as S _XA and the difference in the Z-axis direction is set as S _ZA . Also, the difference S _{XB in} the X-axis direction between the mounting position of the camera B and the position of the rear wheel 612R is set as S _ZB, and the difference in the Z-axis direction is set as S _ZB .

  Here, the turning radius R of the vehicle 601 is obtained by the following equation (2).

The moving direction θ _A of the camera A is obtained by the following equation (3).

The moving direction θ _B of the camera B is obtained by the following equation (4).

Then, from the translation T _{XA in} the X-axis direction and the translation T _{ZA in the} Z-axis direction of the camera A using the movement direction θ _A of the camera _A and the movement direction θ _B of the camera B by the following equation (5), the camera A translation T _{XB in} the X-axis direction of B and a translation T _ZB in the Z-axis direction can be obtained.

And in the monitoring part 201 which detects an obstruction using the image image | photographed with the camera A, it image _| photographed with the camera B using translation _TXA of the X-axis direction of the camera A, and translation _{TZA of} the Z-axis direction. The monitoring unit 201 that detects an obstacle using an image uses the translation T _{XB in} the X-axis direction and the translation T _ZB in the Z-axis direction of the camera B, thereby further improving the obstacle detection accuracy. be able to.

  Note that this behavior parameter correction processing is performed, for example, before sending estimated information to each monitoring unit 201 such as steps S6, S11, S16, and S21 in FIG. 7, step S54 in FIG. 9, and step S107 in FIG. What is necessary is just to make it with respect to the behavior parameter contained in the estimated information to transmit.

  In the above description, an example in which obstacles around the vehicle 131 are detected by the moving stereo method has been described. However, the present invention uses the vehicle behavior or road surface estimation information by a method other than the moving stereo method. The present invention can also be applied to detecting obstacles around the vehicle.

  In addition, the number and installation positions of the cameras 111 described above are examples, and can be changed to an arbitrary number and installation position.

  Furthermore, in the above description, both the behavior parameters and the road surface parameters are estimated and shared using images by the respective monitoring units 201. However, only one of them is used by each monitoring unit 201. May be estimated and shared. For example, each monitoring unit 201 estimates the movement amount of the vehicle 131 from a vehicle speed sensor or the like without estimating the road surface parameter using the image, and only the behavior parameter is estimated using the image by each monitoring unit 201 and shared. You may make it do.

  In the above description, an example in which the monitoring unit 201 is provided for each camera 111 has been described. However, one monitoring unit 201 may be provided for a plurality of cameras 111. In this case, for example, one monitoring unit 201 performs time-series or parallel processing of the own vehicle behavior and road surface estimation for each image captured by the plurality of cameras 111 and the detection of an obstacle.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.

  FIG. 16 is a block diagram illustrating a hardware configuration example of a computer that executes the above-described series of processing by a program.

  In a computer, a CPU (Central Processing Unit) 701, a ROM (Read Only Memory) 702, and a RAM (Random Access Memory) 703 are connected to each other by a bus 704.

  An input / output interface 705 is further connected to the bus 704. An input unit 706, an output unit 707, a storage unit 708, a communication unit 709, and a drive 710 are connected to the input / output interface 705.

  The input unit 706 includes a keyboard, a mouse, a microphone, and the like. The output unit 707 includes a display, a speaker, and the like. The storage unit 708 includes a hard disk, a nonvolatile memory, and the like. The communication unit 709 includes a network interface. The drive 710 drives a removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  In the computer configured as described above, the CPU 701 loads the program stored in the storage unit 708 to the RAM 703 via the input / output interface 705 and the bus 704 and executes the program, for example. Is performed.

  The program executed by the computer (CPU 701) can be provided by being recorded on a removable medium 711 as a package medium, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  In the computer, the program can be installed in the storage unit 708 via the input / output interface 705 by attaching the removable medium 711 to the drive 710. Further, the program can be received by the communication unit 709 via a wired or wireless transmission medium and installed in the storage unit 708. In addition, the program can be installed in advance in the ROM 702 or the storage unit 708.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  Further, in the present specification, the term “system” means an overall apparatus composed of a plurality of apparatuses and means.

  Furthermore, the embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 101 Monitoring system 111N thru | or 111R Camera 112 Gear detection part 113 Steering angle detection part 114 Vehicle speed detection part 115,115A thru | or 115D Image processing part 131 Vehicle 201N thru | or 201R Monitoring part 202 Monitoring control part 211N thru | or 211R Road surface estimation part 212N thru | or 212R Behavior estimation Units 213N to 213R Obstacle detection unit 221 Traveling direction detection unit 222 Setting unit 301 Monitoring control unit 311 Setting unit 401 Monitoring control unit 411 Setting unit 501 Monitoring control unit 511 Traveling direction detection unit 512 Reliability evaluation unit 513 Setting unit

Claims (5)

In an image processing apparatus for detecting an object around the vehicle using a plurality of images taken in different directions by a plurality of cameras provided in the vehicle,
Behavior estimating means for estimating a behavior parameter representing the behavior of the vehicle for each of the plurality of images;
A reliability evaluation that evaluates the reliability of each of the behavior parameters based on at least one of the traveling direction of the vehicle, the number of motion vectors used for estimating the behavior parameters, and the amount of the road surface in the image. Means,
Setting means for setting the behavior parameters used for detecting objects around the vehicle by weighting and averaging the plurality of behavior parameters estimated for each of the plurality of images according to the reliability. ,
An image processing apparatus comprising: object detection means for detecting an object around the vehicle using the behavior parameter set by the setting means and the plurality of images. Road surface estimation means for estimating, for each of the plurality of images, a road surface parameter representing characteristics of a road surface on which the vehicle is traveling,
The setting means further sets the road surface parameter used for detection of objects around the vehicle based on the plurality of road surface parameters estimated for each of the plurality of images.
The image processing apparatus according to claim 1, wherein the object detection unit detects an object around the vehicle using the behavior parameter and the road surface parameter set by the setting unit and the plurality of images. The reliability evaluation unit, the traveling direction of the vehicle, the number of motion vectors used for the estimation of the behavior parameter and the road parameters, and, based on at least one of the amount of the road surface in the image, each of said Evaluate the reliability of the behavior parameters and the road surface parameters ,
The setting means sets the behavior parameter and the road surface parameter used for detecting an object around the vehicle by weighting and averaging a plurality of the behavior parameter and the road surface parameter according to the reliability. The image processing apparatus according to claim 2 . An image processing apparatus that detects an object around the vehicle using a plurality of images taken in different directions by a plurality of cameras provided in the vehicle,
Estimating a behavior parameter representing the behavior of the vehicle for each of the plurality of images,
Evaluating the reliability of each of the behavior parameters based on at least one of the traveling direction of the vehicle, the number of motion vectors used to estimate the behavior parameters, and the amount of road surface in the image;
A plurality of the behavior parameters estimated for each of the plurality of images are weighted and averaged according to the reliability to set the behavior parameters used for detection of objects around the vehicle,
An image processing method including a step of detecting an object around the vehicle using the set behavior parameter and the plurality of images. A computer for detecting objects around the vehicle using a plurality of images taken in different directions by a plurality of cameras provided in the vehicle,
Estimating a behavior parameter representing the behavior of the vehicle for each of the plurality of images,
Evaluating the reliability of each of the behavior parameters based on at least one of the traveling direction of the vehicle, the number of motion vectors used to estimate the behavior parameters, and the amount of road surface in the image;
A plurality of the behavior parameters estimated for each of the plurality of images are weighted and averaged according to the reliability to set the behavior parameters used for detection of objects around the vehicle,
A program for executing a process including a step of detecting an object around the vehicle using the set behavior parameter and the plurality of images.

Images