JP2008003696A - Image processing apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus which is capable of performing discrimination processing of a road plane with less computational complexity. <P>SOLUTION: An image processing apparatus 20 includes: an optical flow calculation means 22 which processes a plurality of pieces of image data picked up at different times to calculate optical flows of respective areas of an image; a flow parameter calculation means 24 which calculates flow parameters of respective areas of the image on the assumption that at least four optical flows calculated by the optical flow calculation means are on one plane; and an area discrimination means 26 which discriminates an area of which the flow parameters calculated by the flow parameter calculation means are similar to flow parameters of a road plane, as an area constituting the road plane. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method for identifying a captured object on an image using an image captured while a camera moves.

Japanese Patent Application Laid-Open No. 2004-133620 proposes a device that mounts a single camera for monitoring the front of a vehicle and processes images captured by the camera to identify surrounding vehicles and obstacles. More specifically, the apparatus according to Patent Document 1 captures the road surface while the camera moves, and superimposes the feature points on the road surface of the first image and the feature points on the road surface of the second image. To calculate a transformation matrix. Then, the apparatus applies this transformation matrix to a plurality of feature points of the image, and identifies feature points that have failed to be superimposed as feature points of a three-dimensional object.
JP 2003-44996 A

  In the above-described device according to the related art, it is necessary to calculate a transformation matrix for superimposing feature points on the road surface and apply this transformation matrix to an image. However, since the amount of calculation for this process is enormous, the calculation cycle and accuracy of the obstacle identification process are limited by the calculation capability of the apparatus. That is, when it is desired to obtain the obstacle identification result with high resolution and high accuracy, it is necessary to delay the calculation period of the obstacle identification processing. On the other hand, when it is desired to obtain an obstacle identification result with a high calculation cycle, the resolution and accuracy of the obstacle identification result is lowered.

  The present invention has been made to solve such a problem, and provides an image processing apparatus and an image processing method capable of identifying a captured object on an image with a small amount of calculation. With the goal.

  In order to achieve the above-described object, an image processing apparatus according to the present invention processes a plurality of image data captured at different times, and calculates an optical flow for each region of the image; Assuming that at least four optical flows calculated by the optical flow calculating means are optical flows on the same plane, the flow parameter calculating means for calculating the flow parameters for each region of the image, and the flow parameter calculating means And a region identifying means for identifying a region similar to the flow parameter of the road plane as a region constituting the road plane. According to this configuration, the image processing apparatus calculates the flow parameter by a relatively simple process, and identifies each region of the image in order to identify the region constituting the road plane of the image using the flow parameter. Therefore, the amount of calculation can be reduced by simplifying the process. Thereby, the resolution of the image in the identification process of each area | region of an image can be improved, or the period which performs the identification process of each area | region of an image can be shortened.

  In the above-described image processing apparatus, the optical flow calculation means calculates a reference optical flow for a characteristic image portion on the road plane, sets a search range for each area of the image based on the reference optical flow, It is preferable to calculate the optical flow for each area of the image by searching inside. According to this configuration, the image processing apparatus sets the search range to an appropriate position in order to set the search range in a part of the image based on the optical flow calculated for the characteristic region on the road plane. And the accuracy of the optical flow can be improved. In particular, by determining the position and area where the search range is set based on the size of the reference optical flow, the search range can be set appropriately, and the accuracy of the optical flow can be improved.

  In the above-described image processing apparatus, it is preferable that the optical flow calculation unit calculates a reference optical flow for a characteristic image portion included in an area in the image in which the traveling direction of the host vehicle is a road plane area. According to this configuration, since the traveling direction of the host vehicle is a road plane in a normal traveling state, the region in the traveling direction of the host vehicle is set as a road plane region in the image, and a characteristic image portion included in this region is determined. A reference optical flow is calculated. Accordingly, the reference optical flow can be set appropriately.

  In the above-described image processing apparatus, the region identifying unit identifies a region in which the flow parameter calculated by the flow parameter calculating unit is similar to the flow parameter of the road plane calculated in the past as a region constituting the road plane. It is preferable to do. According to this configuration, since the image processing apparatus identifies the area constituting the road plane using the flow parameters of the road plane calculated in the past, the area constituting the road plane can be identified by simple processing. Can do.

  In the above-described image processing device, the area identification unit sets the flow parameter of the area as the flow parameter of the road plane when the temporal change of the flow parameter of the area in the traveling direction of the host vehicle is smaller than a preset threshold value. It is preferable. According to this configuration, the image processing apparatus sets the flow parameter of the area as the flow parameter of the road plane when the change over time of the flow parameter of the area in the traveling direction of the host vehicle is small. The road plane in the image can be identified using the flow parameter of the area.

  In the image processing apparatus described above, the area identification unit sets the flow parameter of the area as the flow parameter of the road plane when the flow parameter of the area in the traveling direction of the host vehicle is similar to the flow parameter of the standard road plane. It is preferable to do. According to this configuration, when the flow parameter of the area in the traveling direction of the host vehicle is similar to the flow parameter of the standard road plane, the image processing device sets the flow parameter of the area as the flow parameter of the road plane. Therefore, the road plane in the image can be identified more reliably using the flow parameter of the area that is the road plane.

  In order to achieve the above-described object, an image processing method according to the present invention is a method for calculating an optical flow for each region of an image by processing a plurality of image data captured at different times. Assuming that the calculated at least four optical flows are optical flows on the same plane, a flow parameter calculating step for calculating a flow parameter for each area of the image, and the calculated flow parameter is a road plane And an area identifying step for identifying an area similar to the flow parameter as an area constituting a road plane.

According to the image processing apparatus and the image processing method of the present invention, identification processing of an imaged object on an image can be performed with a small amount of calculation.

  Hereinafter, preferred embodiments of an image processing apparatus of the present invention will be described with reference to the drawings. In the present embodiment, the image processing apparatus is mounted on a vehicle.

  As shown in FIG. 1, a camera 10 is connected to the image processing apparatus 20 as an imaging unit. The camera 10 is, for example, a CCD [Charge Coupled Device] camera 10 and is attached to the front and center of an automobile on which the image processing device 20 is mounted. The camera 10 captures an image of the front of the automobile every time a predetermined time elapses and generates image data composed of a plurality of pixel data. The camera 10 sequentially outputs the generated image data to the image processing device 20.

  When the image processing apparatus 20 captures image data from the camera 10, the image processing apparatus 20 processes the image data to identify an object on the image. In particular, the image processing device 20 identifies whether the object to be imaged on the image is a road plane. The image processing apparatus 20 is physically configured with a CPU, a ROM, a RAM, and the like. Functionally, the image processing apparatus 20 includes an optical flow calculation unit 22, a flow parameter calculation unit 24, and a region identification unit 26. These functions are realized by the CPU and RAM executing programs stored in the ROM.

  The optical flow calculation unit 22 calculates the optical flow vector by processing the image data by the template matching method. The flow parameter calculation unit 24 calculates eight parameters generally called flow parameters based on the calculated optical flow vector. The area identifying unit 26 identifies whether the object to be imaged on the image is a road plane based on the calculated flow parameter.

  A vehicle sensor 30 is connected to the image processing device 20 to enable the above-described processing for identifying an object on the image. The vehicle sensor 30 only needs to detect the behavior of the vehicle. For example, a vehicle speed sensor, a handle angle sensor, a yaw rate sensor, a pitch sensor, a roll sensor, a longitudinal acceleration sensor, a lateral acceleration sensor, a vertical acceleration sensor, etc. May be included. The detection values obtained by these sensors are captured by the image processing apparatus 20 and used for processing for identifying an object to be imaged on the image.

  The result of the object identification processing by the image processing device 20 is used for display control of a monitor provided in the vehicle interior, vehicle operation control, and the like. For example, when the result of the object identification processing by the image processing device 20 is input to the display control device 40, the display control device 40 displays the identification result of each region on the image on the monitor or gives a warning to the driver. Display on the monitor. When the result of the object identification processing by the image processing device 20 is input to the vehicle control device 50, the vehicle control device 50 causes the engine, brake, transmission, steering to avoid collision with an object in front of the vehicle. Control etc.

[Principle of object identification]
Next, the principle by which the image processing apparatus 20 identifies an object on the image will be described.

  First, with reference to FIG. 2, three coordinate systems, that is, an image coordinate system, a camera coordinate system, and a vehicle coordinate system will be described. The image coordinate system is a two-dimensional coordinate system set on the image plane of the camera 10, and the center of the image is the origin, the vertical direction of the image is the x axis, and the horizontal direction of the image is the y axis. The camera coordinate system is a three-dimensional coordinate system set with the camera 10 as the center, the position of the camera 10 is the origin, the vertical direction of the camera 10 is the Xc axis, and the horizontal direction of the camera 10 is the Yc axis. The optical axis direction is taken as the Zc axis. The vehicle coordinate system is a rotation of the camera coordinate system. The vertical direction of the vehicle is the Xv axis, the horizontal direction of the vehicle is the Yv axis, and the traveling direction of the vehicle is the Zv axis. The posture of the camera 10 with respect to the vehicle is shown in FIG. As can be understood from FIG. 3, the angle formed by the Zc axis of the camera coordinate system and the Zv axis of the vehicle coordinate system is θ, and the angle formed by the Yc axis of the camera coordinate system and the Yz axis of the vehicle coordinate system is φ. is there. For convenience of explanation, the angles θ and φ are shown to be large. However, in reality, the camera coordinate system and the vehicle coordinate system almost coincide with each other, and the angles θ and φ are very small. The image coordinate system, the camera coordinate system, and the vehicle coordinate system described above move together with the camera 10 as the vehicle moves.

Next, with reference to FIG. 4, the movement of the camera 10 causing the optical flow will be described. The vehicle translates at the speeds of a, b, and c in the Xv axis direction, Yv axis direction, and Zv axis direction of the vehicle coordinate system, respectively, and ω1, ω2, and ω3 around the Xv axis, the Yv axis, and the Zv axis, respectively. When rotating at an angular velocity, the motion parameters of the camera 10 with respect to the vehicle coordinate system are {a, b, c, ω1, ω2, ω3}. When the motion parameters of the camera 10 based on the vehicle coordinate system are coordinate-converted from the vehicle coordinate system to the camera coordinate system, the motion parameters {a _c , b _c , c _c , ω1 _{c of} the camera 10 based on the camera coordinate system are used. , Ω2 _c , ω3 _c }.

The camera 10 is the above motion parameters _{{a c, b c, c} c, ω1 c, ω2 c, ω3 c} When movement, at a location on the image (x, y), Zc = p · Xc + q · Yc + r When the plane represented by is picked up, an optical flow (u, v) represented by the following equation (1) is caused. Note that u is an x-direction component of the optical flow vector, and v is a y-direction component of the optical flow vector.

  In the above formula (1), f is the focal length of the camera 10. P, q, and r are plane parameters for defining the plane. The eight parameters U, V, A, B, C, D, E, and F are numerical values generally called flow parameters, and are defined by the following formula (2). The mathematical formulas (1) and (2) are described in detail in “Image Understanding—Mathematics of 3D Recognition” (by Kenichi Kanaya).

  As described above, the optical flow is expressed by the mathematical formulas (1) and (2), but is actually calculated by processing a series of images obtained by continuous imaging. That is, an optical flow is calculated by dividing an image at a certain timing into a large number of regions and then searching for regions that match those regions from the image at the next timing by the template matching method.

Furthermore, the flow parameters {U, V, A, B, C, D, E, F} can be calculated from the optical flow calculated in this way. That is, if a part of the area on the image is considered to be a plane, the flow parameters are the same in that area, so information on four or more optical flows in that area (x, y, u, v) From this, the flow parameter can be calculated. For example, four pieces of optical flow information (x ₁ , y ₁ , u ₁ , v ₁ ), (x ₂ , y ₂ , u ₂ , v ₂ ), (x ₃ , y ₃ , When u ₃ , v ₃ ), (x ₄ , y ₄ , u ₄ , v ₄ ) and the focal length f of the camera 10 are substituted into the above equation (1), flow parameters {U, V, A, B, A total of eight mathematical expressions including C, D, E, F} as unknowns are obtained. Therefore, the numerical values of the flow parameters can be obtained by processing these eight mathematical expressions by a known method. When five or more pieces of optical flow information are used for calculating the flow parameter, an approximation method such as a least square method may be used. In the following description, the flow parameters calculated for the position (x, y) on the image are represented by {U _{x, y} , V _{x, y} , A _{x, y} , B _{x, y} , C _{x, y} , D _{x, y} , _{Ex, y} , _{Fx, y} }.

As can be understood from the above formula (2), the flow parameters {U, V, A, B, C, D, E, F} are the motion parameters {a _c , b _c , c _c , ω 1 _c of the camera 10. , Ω2 _c , ω3 _c } and the plane parameters {p, q, r}. Therefore, the motion parameters of the camera _{10 {a c, b c,} c c, ω1 c, ω2 c, ω3 c} When is a constant value, the flow parameters {U, V, A, B , C, D, E, F} can be used as a numerical value equivalent to the planar parameter {p, q, r} of the imaged object surface. In the principle of object identification in the present embodiment, such characteristics of flow parameters are used, and the plane of each region on the image is identified based on the flow parameter calculated for each region on the image.

In other words, the eight flow parameters have substantially the same value for the image areas obtained by imaging the same plane. Therefore, the flow parameters { _{Ux, y} , Vx _{, y} , _{Ax, y} , _{Bx, y} , _{Cx, y} , _{Dx, y} , _{Ex, y} , _{Fx, y} of each region on the image } With each other, it can be determined whether or not the two image regions are on the same plane. In particular, in the present embodiment, the flow parameters {U _r , V _r , A _r , B _r , of the plane based on the road plane are used as the flow parameters of the reference plane whose geometrical arrangement in the three-dimensional coordinate system is known. C _r , D _r , E _r , F _r } are obtained. Then, by comparing the flow parameter of each area on the image with the flow parameter of the road plane, it is identified whether each area on the image is a road plane. Here, when both flow parameters are similar, the region on the image is identified as a road plane. The similarity between both flow parameters is when both flow parameters match and when the difference between both flow parameters is smaller than a predetermined threshold.

[Object identification processing]
Next, the object identification process by the image processing apparatus 20 according to the present embodiment will be described with reference to the flowcharts of FIGS. FIG. 5 shows an overall flow of object identification processing by the image processing apparatus 20. FIG. 6 shows a flow of processing for calculating an optical flow. FIG. 7 shows a flow of processing for setting a road plane flow parameter. FIG. 8 shows another processing flow for setting the flow parameter of the road plane.

As shown in the flowchart of FIG. 5, the image processing apparatus 20 takes in the image data to the timing _{T n} for each predetermined time interval (e.g., 30 msec) (S501). An example of image data captured by the image processing apparatus 20 is shown in FIG. In this image, the road extends forward. There are oncoming vehicles on the road, and a sidewalk extends to the end of the road. The image processing device 20 processes the captured image data with a LoG filter (Laplacian of Gaussian Filter). Next, the image processing apparatus 20 divides the entire image surface into a rectangular small area (for example, an 8 × 8 pixel area, hereinafter referred to as a small area) in a mesh pattern (see FIG. 10), and performs template matching. An optical flow vector is calculated for each small region (S502, see FIG. 11). The process of calculating the optical flow will be described in detail later with reference to FIG.

Next, the image processing apparatus 20 sets a region including four adjacent small regions as a set of regions (hereinafter referred to as a middle region), and assumes that each middle region is a single plane. The middle region flow parameters are calculated (S503). Optical flow information (x ₁ , y ₁ , u ₁ , v ₁ ), (x ₂ , y ₂ , u ₂ , v ₂ ), (x ₃ , y ₃ , u ₃ ) of four small regions included in the middle region , V ₃ ), (x ₄ , y ₄ , u ₄ , v ₄ ), the flow parameters {U _{x, y} , V _{x, y} , A _{x, y} , B _{x, y} , C _{x, y} , D _{x, y} , E _{x, y} , F _{x, y} } are calculated as described above.

Next, the image processing apparatus 20 reliably sets the flow parameters {U _r , V _r , A _r , B _r , C _r , D _r , E _r , F _r } in the middle region that is the road plane (S504). ). The process of setting the road plane flow parameter will be described in detail later with reference to FIGS. Then, the image processing apparatus 20, the flow parameters _{{U x} of the middle region of the _{image, y, V x, y,} A x, y, B x, y, C x, y, D x, y, E x _{, Y} , F _{x, y} } are compared with the flow parameters {U _r , V _r , A _r , B _r , C _r , D _r , E _r , F _r } on the road plane. It is determined whether or not to perform (S505). Specifically, the image processing apparatus 20 obtains a difference ΔFP _r between both flow parameters using the following formula (3.1) or formula (3.2), and the difference ΔFP _r is set to a preset threshold TH. It is judged whether it is larger than.

Here, k _U , k _V , k _A , k _B , k _C , k _D , k _E , and k _F are weighting factors for each flow parameter. As the weighting factors k _{U to} k _F of each flow parameter, a value suitable for identifying an area corresponding to the road plane may be set. Here, different values may be set as the weighting coefficients k _{U to} k _F of the flow parameters, or common values may be set. The threshold value TH to be compared with the difference ΔFP _r is set to a value suitable for identifying an area corresponding to the road plane. For example, the threshold value TH may be a different value depending on the traveling speed of the host vehicle and the distance from the camera 10 to the imaging position on the road.

If it is determined in step 505 that the difference ΔFP _r is smaller than the threshold value TH, the flow parameter of the middle area is similar to the flow parameter of the road plane, so that the middle area is identified as the road plane (S506). ). For example, an area indicated by diagonal lines in FIG. 12 is identified as a road plane. On the other hand, if it is determined in step 505 that the difference ΔFP _r is larger than the threshold value TH, the flow parameter of the middle area is not similar to the flow parameter of the road plane, and therefore the middle area is identified as other than the road plane. To do.

  Next, processing for calculating an optical flow will be described with reference to FIG. FIG. 6 shows a flowchart of processing for calculating an optical flow. The flowchart shown in FIG. 6 corresponds to step 502 in FIG.

In step 601 to step 607, the image processing apparatus 20 calculates a reference optical flow serving as a reference as an optical flow on the road plane. First, the image processing apparatus 20 extracts a small area including a characteristic image portion from an area presumed to be a road plane in the image captured at time T _n (S601). ). The small area extracted by the image processing device 20 is an area including characteristic image portions such as a pedestrian crossing, a white line, and a road sign drawn on the road plane. In the processing example shown in FIG. 13, the edge part of the pedestrian crossing within the circle is extracted as a characteristic image part from the region corresponding to the road plane of the image. FIG. 14 is an enlarged view of a small area including a characteristic image portion. Such a small region including a characteristic image portion is suitable for pattern matching processing, and an optical flow vector can be calculated with high accuracy by using this region.

  Since there are a plurality of characteristic image portions in the region estimated to be a road plane, the image processing apparatus 20 may extract a plurality of small regions including these characteristic image portions. However, the image processing device 20 needs to avoid extracting a small region corresponding to another vehicle traveling ahead. For this purpose, the image processing apparatus 20 stores in advance standard inter-vehicle distance data for each vehicle speed. Then, the image processing device 20 takes in the detected value of the vehicle speed from the vehicle sensor 30 and extracts a characteristic image portion of the road plane from the area in front of the vehicle and within the inter-vehicle distance corresponding to the vehicle speed. It is preferable to extract a small area to be included.

Next, the image processing apparatus 20 sets the center (x _n , y _n ) of the small region including the characteristic image portion as the start point of the optical flow vector (S602). Furthermore, as shown in FIG. 14, the image processing apparatus 20 displays a characteristic image portion in a range that is the same as the center of the small region including the characteristic image portion in the image captured at the time T _n. A range that is slightly larger than the included small region is determined as a search range to be subjected to template matching processing. Then, the image processing device 20 sets a search range at the same position in the image coordinate system in the image captured at time _{Tn + 1} (S603, see FIG. 15).

Next, the image processing apparatus 20 uses a small area including the characteristic image portion of the time T _n as a template, performing template matching processing for the set search range at time T _{n + 1} of the image, time A correlation value is calculated for each small region included in the search range at T _{n + 1} (S604). Then, the image processing apparatus 20 compares the correlation values calculated for each small region with each other to obtain a small region with the maximum correlation value (S605). As a result, as shown in FIG. 15, the small region compatible small region of the image of the timing T _n is searched from the search range of the time T _{n + 1} of the image.

Next, the image processing apparatus 20 sets the center (x _{n + 1} , y _{n + 1} ) of the small region including the characteristic image portion in the image at time T _{n + 1} as the end point of the optical flow vector (S606). Then, the image processing apparatus 20 calculates a vector from the start point to the end point of the optical flow vector as a reference optical flow vector (x _{n + 1} −x _n , y _{n + 1} −y _n ) serving as a reference as an optical flow on the road plane ( S607).

Next, the image processing apparatus 20 sets the search range for each small region on the image while adjusting the search range based on the optical flow vector of the characteristic image portion calculated in step 607 (S608). . Specifically, the image processing apparatus 20 first applies, for each of the small regions on the image, a range that is the same as the small region and has a center that is slightly larger than the small region. Is temporarily set as a search range. Then, each of the temporarily set search ranges is moved according to the distance and direction of the reference optical flow vector (x _{n + 1} −x _n , y _{n + 1} −y _n ).

  FIG. 16 shows a search range that is set when the vehicle speed is high. When the vehicle speed is high, the reference optical flow vector is large, and therefore the search range is set to a position (solid line) that is largely shifted from the initially set position (dashed line). On the other hand, FIG. 17 shows a search range set when the vehicle speed is low. When the vehicle speed is low, the reference optical flow vector is small, so the search range is set at a position (solid line) close to the initially set position (broken line). By adjusting the position of the search range in this way, the small region after movement surely enters the search range, so that the small region after movement can be reliably searched, and the optical flow vector is calculated with high accuracy. It becomes possible.

  In setting the search range, the area of the search range may be adjusted according to the reference optical flow vector. For example, the area of the search range may be increased when the reference optical flow vector is large, and the area of the search range may be decreased when the reference optical flow vector is small. By adjusting the area of the search range in this way, the search range can be adjusted to the minimum necessary size, so that the calculation time of the optical flow vector calculation process can be shortened. Further, by adjusting the search range to the minimum necessary size, the probability that an erroneous small area is searched is reduced, so that the optical flow vector can be calculated with high accuracy.

Finally, the image processing apparatus 20, with each small area of the time T _n of the image as a template by performing a template matching process with respect to the set search range at time T _{n + 1} of the image, the time T _{n + 1} of the image A small region having the maximum correlation value is obtained from the search range set to. Then, the image processing apparatus 20 starting from the center of each small area at time T _n, the vector and ending the center of each small region after movement at time T _{n + 1,} is calculated as the optical flow vector for each area (S609).

  Next, with reference to FIG. 7, a process for setting the flow parameters of the road plane will be described. FIG. 7 shows a flowchart of a process for setting a flow parameter for a road plane. The flowchart shown in FIG. 7 corresponds to step 504 in FIG.

First, the image processing apparatus 20 estimates that a plurality of middle regions in the vehicle traveling direction correspond to the road plane, monitors the flow parameters calculated for these middle regions, and determines the magnitude of the change over time of the flow parameters. Is obtained (S701). Here, the magnitude of the change over time is determined by the flow parameters {U _{x, y} , V _{x, y} , A _{x, y} , B _{x, y} , C _{x, y} , D _{x, y} , E _{x, y} , F _{x. , Y} }. Next, the image processing apparatus 20 compares the magnitude of the change over time of each flow parameter with preset thresholds T _U , T _V , T _A , T _B , T _C , T _D , T _E , and T _F. Then, a middle region in which the magnitude of the flow parameter change with time is smaller than the thresholds T _{U to} T _F is determined (S702).

Here, the thresholds T _{U to} T _F are for determining the degree of stability of the flow parameter. If the middle area continues to be a road plane while the flow parameter change over time is measured for a middle area, the middle area flow parameter is stable and the flow parameter changes over time. The magnitude is smaller than the thresholds T _{U to} T _F. On the other hand, when the middle area is other than the road plane, the flow parameter of the middle area changes and becomes unstable, and the magnitude of the change of the flow parameter with time becomes larger than the thresholds T _{U to} T _F. Therefore, when the magnitude of the change over time of the flow parameter is smaller than the threshold values T _{U to} T _F , it is possible to determine that the middle region is a road plane.

However, depending on the running state of the vehicle, angular velocity changes ω1, ω2, and ω3 may occur in the vehicle. Thus, when the angular velocity changes ω1, ω2, and ω3 occur, the angular velocity changes ω1, ω2, and ω3 are included in Equation (2) even though the middle region of the vehicle traveling direction actually corresponds to the road plane. As the flow parameter changes with time according to the change in the angular velocity, it becomes impossible to determine that the middle region in the vehicle traveling direction corresponds to the road plane. Therefore, when the vehicle is in a traveling state in which the angular velocity changes ω1, ω2, and ω3 are generated, a flow parameter that does not include the angular velocity change in Equation (2) is selected, and the temporal change of the selected flow parameter and the threshold value T _U. Compare _TF .

In actual processing, the image processing apparatus 20 selects the flow parameters A, B, C, and D, and changes the flow parameters A, B, C, and D over time with the thresholds T _A , T _B , T _C , and T _D. It is preferable to compare with. Here, the flow parameters A and D do not include the yaw rate ω1, the pitch rate ω2, and the roll rate ω3 in the equation (2). Further, the flow parameters B and C do not include the yaw rate ω1 and the pitch rate ω2 in the equation (2). Therefore, even in a traveling state in which the yaw rate ω1 and the pitch rate ω2 are generated, the change over time in the flow parameters A, B, C, and D is smaller than the threshold values T _A , T _B , T _C , and T _D. It can be determined that the middle area of the direction corresponds to the road plane. Although the flow parameters B and C include the roll rate ω3 in the formula (2), since the value of the roll rate generated in the vehicle is small, the change over time of the flow parameters B and C is small.

In step 703, the image processing apparatus 20 determines the flow parameters { _{Ux, y} , Vx _{, y} , _{Ax, y} , _{Bx, y} , _{Cx, y} , _{Dx, y} , Compare E _{x, y} , F _{x, y} } with standard road plane flow parameters {U _av , V _av , A _av , B _av , C _av , D _av , E _av , F _av } A middle region where both flow parameters are similar is determined (S703). Here, the standard road plane flow parameter is a road plane flow parameter generated when the camera 10 is placed at a standard position and orientation as shown in FIG. This is a value obtained in advance. Specifically, the image processing apparatus 20 obtains a difference ΔFP _av between the flow parameters using the following formula (4.1) or formula (4.2), and the difference ΔFP _av is set to a preset threshold value TH. A middle region that is smaller than is determined.

In step 703, if the difference FP _av between both flow parameters is smaller than a preset threshold value TH, the flow parameter in the middle region of the vehicle traveling direction is a reasonable value for the road plane. Therefore, the image processing apparatus 20 sets the flow parameter of the middle region in the vehicle traveling direction as the flow parameter of the road plane (S704).

  Next, with reference to FIG. 8, a modified example of the process for setting the flow parameter of the road plane will be described. FIG. 8 shows a flowchart of the process for setting the flow parameter of the road plane. The flowchart shown in FIG. 8 corresponds to step 504 in FIG.

First, the image processing apparatus 20 is the flow parameter calculated in step 503, which is a flow parameter {U _n for each middle region of an image captured at a time T _n−k before a predetermined time (for example, 3 seconds before). _{-k, V n-k, a} n-k, B n-k, C n-k, D n-k, E n-k, to obtain the _{F n-k} (S801)} . Next, the image processing apparatus 20 also uses the flow parameters {U _n , V _n , A _n , B _{n for the} middle region of the image captured at the current time T _n , which are the flow parameters similarly calculated in step 503. , C _n , D _n , E _n , F _n } are acquired (S802).

Then, the image processing device 20 determines the flow parameters {U _n , V _n , A _n , B _n , C _n , D _n , E _n , F _n } at the current time T _n for each middle region of the image. Flow parameters {U _n−k , V _n−k , A _n−k , B _n−k , C _n−k , D _n−k , E _n−k , F _n− at time T _n−k before time _k }, a middle region in which both flow parameters are similar is discriminated (S803). Specifically, the image processing apparatus 20 obtains a difference ΔFP _n between both flow parameters using the following formula (5.1) or formula (5.2), and the difference ΔFP _n is set to a preset threshold value TH. A middle region that is smaller than is determined.

The flow parameters in the middle area corresponding to buildings and other vehicles change significantly. On the other hand, the flow parameter in the middle region corresponding to the road plane hardly changes. Therefore, when the difference FP _n between the flow parameters is smaller than the preset threshold value TH in step 703, it is highly possible that the middle region where the change in the flow parameters is small is a road plane. Therefore, the image processing apparatus 20 sets the flow parameter for the middle region where the difference FP _n is smaller than the threshold value TH as the flow parameter for the road plane (S804).

FIG. 18 shows an image at time T _n−k before a predetermined time, and FIG. 19 shows an image at current time T _n . In FIG. 18 and FIG. 19, the middle region surrounded by the thick line corresponds to the road plane at both times T _n−k and T _n . In this way, the flow parameter of the middle region obtained by imaging the road plane at both times T _n−k and T _n is set as the flow parameter of the road plane. As shown in FIGS. 18 and 19, when there are a plurality of middle areas obtained by imaging the road plane at both times T _n−k and T _n , the average value of the flow parameters of these middle areas is expressed as What is necessary is just to set as a flow parameter of a road plane.

  The road plane flow parameters set by the processing of FIG. 7 or FIG. 8 can be used repeatedly in the subsequent processing. That is, when the image processing apparatus calculates the flow parameter for each middle region of the image, the flow parameter calculated for each middle region is compared with the flow parameter of the road plane calculated in the past. It is preferable to identify the region to be configured. According to this configuration, since the flow parameter of the road plane has already been calculated, the image processing apparatus can identify the area constituting the road plane with a simple process.

It is the schematic which shows the image processing apparatus of this embodiment. It is a schematic diagram for demonstrating an image coordinate system, a camera coordinate system, and a vehicle coordinate system. It is a schematic diagram for demonstrating the attitude | position of the camera mounted in the vehicle. It is a schematic diagram for demonstrating the motion parameter of a vehicle. It is a flowchart which shows the outline of an object identification process. It is a flowchart which shows the calculation process of an optical flow. It is a flowchart which shows the process which sets the flow parameter of a road plane. It is another flowchart which shows the process which sets the flow parameter of a road plane. It is a figure which shows the image taken in by the image processing apparatus. It is a figure which shows the small area | region of an image. It is a figure which shows the optical flow of an image. It is a figure which shows the road plane identified from the image. It is a figure which shows the characteristic image part extracted from an image. It is a figure which expands and shows the characteristic image part. It is a figure which expands and shows the characteristic image part after a movement. It is a figure which shows the search range set when a vehicle speed is large. It is a figure which shows the search range set when a vehicle speed is small. It is a figure which shows the area | region corresponding to a road plane in time _Tn-k . It is a diagram showing a region corresponding to the road plane at time T _n.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Camera, 20 ... Image processing apparatus, 30 ... Vehicle sensor, 40 ... Display control apparatus, 50 ... Vehicle operation control apparatus.

Claims (9)

An optical flow calculation means for processing a plurality of image data captured at different times and calculating an optical flow for each area of the image;
Assuming that at least four optical flows calculated by the optical flow calculating means are optical flows on the same plane, a flow parameter calculating means for calculating a flow parameter for each region of the image;
An area identifying means for identifying an area where the flow parameter calculated by the flow parameter calculating means is similar to the flow parameter of the road plane as an area constituting the road plane;
An image processing apparatus comprising:   The optical flow calculation means calculates a reference optical flow for a characteristic image portion on a road plane, sets a search range for each region of the image based on the reference optical flow, and searches within the search range. The image processing apparatus according to claim 1, wherein an optical flow for each region of the image is calculated.   3. The optical flow calculation unit according to claim 2, wherein an area in the traveling direction of the host vehicle in the image is a road plane area, and a reference optical flow is calculated for a characteristic image portion included in the area. Image processing apparatus.   The image processing apparatus according to claim 2, wherein the optical flow calculation unit determines a position where the search range is set based on a size of the reference optical flow.   The image processing apparatus according to claim 2, wherein the optical flow calculation unit determines an area in which the search range is set based on a size of the reference optical flow.   The area identifying means identifies an area in which the flow parameter calculated by the flow parameter calculating means is similar to a flow parameter of a road plane calculated in the past as an area constituting a road plane. The image processing apparatus according to claim 1.   The area identifying means sets the flow parameter of the area as the flow parameter of the road plane when the change over time of the flow parameter of the area in the traveling direction of the host vehicle is smaller than a preset threshold value. The image processing apparatus according to claim 1.   The area identification means sets the flow parameter of the area as the flow parameter of the road plane when the flow parameter of the area in the traveling direction of the own vehicle is similar to the flow parameter of the standard road plane. The image processing apparatus according to claim 1. An optical flow calculation step of processing a plurality of image data captured at different times and calculating an optical flow for each region of the image;
Assuming that the calculated at least four optical flows are optical flows on the same plane, a flow parameter calculating step for calculating a flow parameter for each region of the image;
An area identifying step for identifying an area in which the calculated flow parameter is similar to the flow parameter of the road plane as an area constituting the road plane;
An image processing method including:

Images