WO2010116801A1

WO2010116801A1 - Image processing device

Info

Publication number: WO2010116801A1
Application number: PCT/JP2010/052530
Authority: WO
Inventors: 洋平石井; 亮平山本
Original assignee: 三洋電機株式会社
Priority date: 2009-04-06
Filing date: 2010-02-19
Publication date: 2010-10-14
Also published as: JP2010245803A

Abstract

When focusing on a camera (C_3) and another camera (C_4) among cameras (C_1 to C_4), the camera (C_3) and the other camera (C_4) partially have a common visual field (VW_34) and capture a field image in a direction that intersects the road surface in a slanting manner. A CPU takes in a field image (P_3) and another field image (P_4) outputted from the camera (C_3) and the other camera (C_4) respectively, and creates a birds eye image representing the state of capture of the field from above the road surface, based on the field image (P_3) and the other field image (P_4) that were taken in. Further, when there is an obstacle in the common visual field (VW_34), the CPU adjusts the posture of the obstacle image on the birds eye image, based on the positional relationship of the camera (C_3) and the other camera (C_4) with the obstacle.

Description

Image processing device

The present invention relates to an image processing apparatus, and in particular, captures a scene from above a reference plane based on a scene image output from a camera that captures the scene in a direction obliquely intersecting the reference plane. The present invention relates to an image processing apparatus that creates a bird's-eye view image representing a state.

An example of this type of device is disclosed in Patent Document 1. According to this background art, the four cameras are mounted on the vehicle so that the fields of view of the two adjacent cameras are partially common. The surroundings of the vehicle are captured by such a camera, and the object scene image output from each camera is converted into a bird's-eye view image.

Boundary lines are assigned to partial images corresponding to the common visual field in the converted bird's-eye view image. The four bird's-eye images respectively corresponding to the four cameras are combined with each other through the trimming process referring to such a boundary line. However, when an obstacle (three-dimensional object) is detected from the common visual field, the position of the boundary line is changed so as to avoid the obstacle image. Thereby, the quality of the obstacle image is maintained.

JP 2007-104373 A

However, when an obstacle is captured with a field of view common to two adjacent cameras, due to the nature of the bird's-eye view image, the posture of the obstacle image that appears in the bird's-eye image corresponding to one camera is the bird's-eye view corresponding to the other camera. It is different from the posture of the obstacle image that appeared in the image. For this reason, if the camera which catches the obstacle which appeared in the synthetic bird's-eye view image is changed along with the change of the boundary line, the posture of the obstacle image changes discontinuously.

Therefore, a main object of the present invention is to provide an image processing apparatus capable of improving the visibility of a three-dimensional object in a common visual field.

An image processing apparatus according to the present invention includes the following: a plurality of cameras each having a common field of view and capturing an object scene in a direction obliquely intersecting a reference plane; respectively output from a plurality of cameras Creation means for creating a bird's-eye view image with respect to a reference plane based on a plurality of object scene images; discrimination means for discriminating a positional relationship between a three-dimensional object existing in a common visual field and a plurality of cameras; Adjustment means for adjusting the posture of the three-dimensional object image on the image based on the determination result of the determination means.

Preferably, the determining means determines whether or not the three-dimensional object is sandwiched by a plurality of reference lines extending in a predetermined manner from the plurality of cameras toward the object scene, and the adjusting means is a three-dimensional object along the plurality of reference lines. Adjust the posture of the image.

In one aspect, the default mode corresponds to a mode in which a plurality of reference lines are parallel to each other.

In another aspect, the three-dimensional object image corresponds to a bird's-eye image of a three-dimensional object captured by a reference camera that is one of a plurality of cameras, and the adjustment unit belongs to a field of view sandwiched by a plurality of reference lines. Definition means for defining a line connecting the reference camera and the three-dimensional object, and correction means for correcting the posture of the three-dimensional object image with reference to a difference in angle between the line defined by the definition means and the reference line.

More preferably, the adjustment means includes a selection means for selecting, as a reference camera, a camera closer to the three-dimensional object among the plurality of cameras when the three-dimensional object deviates from the field of view sandwiched by the plurality of reference lines.

Preferably, a restricting means for restricting the adjustment process of the adjusting means when the three-dimensional object meets a specific condition is further provided.

According to the present invention, the posture of the three-dimensional object image on the bird's-eye view image is adjusted based on the positional relationship between the three-dimensional object existing in the common visual field and the plurality of cameras. Thereby, the change in the posture of the three-dimensional object image due to the change in the positional relationship between the three-dimensional object and the camera is suppressed, and the visibility of the three-dimensional object in the common visual field is improved.

The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

It is a block diagram which shows the basic composition of this invention. It is a block diagram which shows the structure of one Example of this invention. It is an illustration figure which shows the visual field captured by the some camera attached to the vehicle. (A) is an illustrative view showing an example of a bird's eye view image based on the output of the previous camera, (B) is an illustrative view showing an example of a bird's eye view image based on the output of the right camera, and (C) is an output of the left camera. It is an illustration figure which shows an example of the bird's-eye view image based on (D), and (D) is an illustration figure which shows an example of the bird's-eye view image based on the output of a back camera. It is an illustration figure which shows a part of creation operation of an all-around bird's-eye view image. It is an illustration figure which shows an example of the created all-around bird's-eye view image. It is an illustration figure which shows an example of the driving assistance image displayed by a display apparatus. It is an illustration figure which shows the angle of the camera attached to the vehicle. It is an illustration figure which shows the relationship between a camera coordinate system, the coordinate system of an imaging surface, and a world coordinate system. It is an illustration figure which shows an example of the obstruction which exists in a vehicle and its periphery. FIG. 10 is an illustrative view showing another portion of the operation of creating the driving assistance image. It is an illustration figure which shows an example of the operation | movement which reproduces an obstacle image. It is an illustration figure which shows another example of the operation | movement which reproduces an obstacle image. It is an illustration figure which shows another example of the operation | movement which reproduces an obstacle image. (A) is an illustrative view showing a reproduction state of an obstacle image before posture adjustment, and (B) is an illustrative view showing a reproduction state of the obstacle image after posture adjustment. It is a flowchart which shows a part of operation | movement of CPU applied to the FIG. 2 Example. It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is an illustration figure which shows another example of the allocation state of an adjustment area. It is an illustration figure which shows another example of the allocation state of an adjustment area. It is an illustration figure which shows another example of the allocation state of an adjustment area. It is a flowchart which shows a part of operation | movement of CPU applied to another Example.

Referring to FIG. 1, the image processing apparatus of the present invention is basically configured as follows. Each of the plurality of

cameras

1, 1,... Has a common field of view and captures an object scene in a direction that obliquely intersects the reference plane. The creating unit 2 creates a bird's-eye view image with respect to the reference plane based on a plurality of object scene images respectively output from the plurality of

cameras

1, 1,. The discriminating means 3 discriminates the positional relationship between the three-dimensional object existing in the common visual field and the plurality of

cameras

1, 1,. The adjusting unit 4 adjusts the posture of the three-dimensional object image on the bird's-eye image created by the creating unit 2 based on the determination result of the determining unit 3.

The camera 1 captures the object scene in a direction that crosses the reference plane obliquely. For this reason, when a three-dimensional object exists on the reference plane, the three-dimensional object image appears in the bird's-eye view image in a posture of falling down along a line connecting the three-dimensional object and the camera 1. Further, the posture of the three-dimensional object image changes due to a change in the positional relationship between the three-dimensional object and the camera 1.

According to this invention, the posture of the three-dimensional object image on the bird's-eye view image is adjusted based on the positional relationship between the three-dimensional object existing in the common visual field and the plurality of

cameras

1, 1,. Thereby, the change in the posture of the three-dimensional object image due to the change in the positional relationship between the three-dimensional object and the camera 1 is suppressed, and the visibility of the three-dimensional object in the common visual field is improved.

The steering support device 10 of this embodiment shown in FIG. 2 includes four cameras C_1 to C_4. The cameras C_1 to C_4 output scene images P_1 to P_4 every 1/30 seconds in synchronization with the common timing signal. The output scene images P_1 to P_4 are given to the image processing circuit 12.

Referring to FIG. 3, camera C_1 is installed at the center of the front portion of vehicle 100 in a posture in which the optical axis of camera C_1 extends obliquely downward in front of vehicle 100. The camera C_2 is installed at the center on the right side of the vehicle 100 with the optical axis of the camera C_2 extending obliquely downward to the right of the vehicle 100. The camera C_3 is installed on the upper rear side of the vehicle 100 in a posture in which the optical axis of the camera C_3 extends obliquely downward rearward of the vehicle 100. The camera C_4 is installed at the center of the left side of the vehicle 100 in a posture in which the optical axis of the camera C_4 extends obliquely downward to the left of the vehicle 100. The object scene around the vehicle 100 is captured from such a direction that intersects the road surface in an oblique direction by the cameras C_1 to C_4.

Camera C_1 has a field of view VW_1 that captures the front of the vehicle 100, camera C_2 has a field of view VW_2 that captures the right direction of the vehicle 100, camera C_3 has a field of view VW_3 that captures the rear of the vehicle 100, and camera C_4 It has a visual field VW_4 that captures the left direction of the vehicle 100. Also, the visual fields VW_1 and VW_2 have a common visual field VW_12, the visual fields VW_2 and VW_3 have a common visual field VW_23, the visual fields VW_3 and VW_4 have a common visual field VW_34, and the visual fields VW_4 and VW_1 have a common visual field VW_41.

Returning to FIG. 2, the CPU 12p provided in the image processing circuit 12 generates the bird's-eye view image BEV_1 shown in FIG. 4A based on the object scene image P_1 output from the camera C_1, and is output from the camera C_2. The bird's-eye view image BEV_2 shown in FIG. 4B is generated based on the object scene image P_2. The CPU 12p also generates the bird's-eye view image BEV_3 shown in FIG. 4C based on the object scene image P_3 output from the camera C_3, and based on the object scene image P_4 output from the camera C_4, FIG. The bird's-eye view image BEV_4 shown in FIG.

The bird's-eye view image BEV_1 corresponds to an image captured by a virtual camera looking down the visual field VW_1 in the vertical direction, and the bird's-eye view image BEV_2 corresponds to an image captured by a virtual camera looking down the visual field VW_2 in the vertical direction. The bird's-eye view image BEV_3 corresponds to an image captured by a virtual camera looking down the visual field VW_3 in the vertical direction, and the bird's-eye view image BEV_4 corresponds to an image captured by a virtual camera looking down the visual field VW_4 in the vertical direction.

According to FIGS. 4A to 4D, the bird's-eye image BEV_1 has a bird's-eye coordinate system X1 and Y1, the bird's-eye image BEV_2 has a bird's-eye coordinate system X2 and Y2, and the bird's-eye image BEV_3 has a bird's-eye coordinate system. The bird's-eye view image BEV_4 has a bird's-eye coordinate system X4 / Y4. Such bird's-eye images BEV_1 to BEV_4 are held in the work area W1 of the memory 12m.

Subsequently, the CPU 12p deletes a part of the images outside the boundary line BL from each of the bird's-eye images BEV_1 to BEV_4, and rotates / moves the bird's-eye images BEV_1 to BEV_4 (see FIG. 5) remaining after the deletion. Join each other. As a result, the all-around bird's-eye view image shown in FIG. 6 is obtained in the work area W2 of the memory 12m.

In FIG. 6, the overlapping area OL_12 indicated by diagonal lines corresponds to an area that reproduces the common visual field VW_12, and the overlapping area OL_23 indicated by diagonal lines corresponds to an area that reproduces the common visual field VW_23. Further, the overlapping area OL_34 indicated by hatching corresponds to an area for reproducing the common visual field VW_34, and the overlapping area OL_41 indicated by hatching corresponds to an area for reproducing the common visual field VW_41.

The display device 16 installed in the driver's seat extracts a part of the images D1 in which the overlapping areas OL_12 to OL_41 are located at the four corners from the all-around bird's-eye view image on the work area W2, and the vehicle image D2 imitating the upper part of the vehicle 100 Is pasted in the center of the extracted image D1. As a result, the driving assistance image shown in FIG. 7 is displayed on the monitor screen.

Next, how to create the bird's-eye view images BEV_1 to BEV_4 will be described. However, since all the bird's-eye images BEV_1 to BEV_4 are created in the same manner, the creation procedure of the bird's-eye image BEV3 will be described as a representative of the bird's-eye images BEV_1 to BEV_4.

Referring to FIG. 8, camera C_3 is disposed rearward and obliquely downward at the rear of vehicle 100. If the depression angle of the camera C_3 is “θd”, the angle θ shown in FIG. 7 corresponds to “180 ° −θd”. Further, the angle θ is defined in the range of 90 ° <θ <180 °.

FIG. 9 shows the relationship between the camera coordinate system X · Y · Z, the coordinate system Xp · Yp of the imaging surface S of the camera C_3, and the world coordinate system Xw · Yw · Zw. The camera coordinate system X, Y, Z is a three-dimensional coordinate system with the X, Y, and Z axes as coordinate axes. The coordinate system Xp · Yp is a two-dimensional coordinate system having the Xp axis and the Yp axis as coordinate axes. The world coordinate system Xw · Yw · Zw is a three-dimensional coordinate system having the Xw axis, the Yw axis, and the Zw axis as coordinate axes.

In the camera coordinate system X, Y, Z, the optical center of the camera C3 is defined as the origin O, the Z axis is defined in the optical axis direction, the X axis is defined in the direction perpendicular to the Z axis and parallel to the road surface, and A Y axis is defined in a direction orthogonal to the Z axis and the X axis. In the coordinate system Xp / Yp of the imaging surface S, with the center of the imaging surface S as the origin, the Xp axis is defined in the horizontal direction of the imaging surface S, and the Yp axis is defined in the vertical direction of the imaging surface S.

In the world coordinate system Xw / Yw / Zw, the intersection of the vertical line passing through the origin O of the camera coordinate system XYZ and the road surface is defined as the origin Ow, and the Yw axis is defined in the direction perpendicular to the road surface. An Xw axis is defined in a direction parallel to the X axis of Z, and a Zw axis is defined in a direction orthogonal to the Xw axis and the Yw axis. The distance from the Xw axis to the X axis is “h”, and the obtuse angle formed by the Zw axis and the Z axis corresponds to the angle θ described above.

When coordinates in the camera coordinate system X, Y, and Z are expressed as (x, y, z), “x”, “y”, and “z” are X-axis components in the camera coordinate system X, Y, and Z, respectively. A Y-axis component and a Z-axis component are shown. When the coordinates in the coordinate system Xp / Yp of the imaging surface S are expressed as (xp, yp), “xp” and “yp” respectively represent the Xp-axis component and the Yp-axis component in the coordinate system Xp / Yp of the imaging surface S. Show. When coordinates in the world coordinate system Xw · Yw · Zw are expressed as (xw, yw, zw), “xw”, “yw”, and “zw” are Xw axis components in the world coordinate system Xw · Yw · Zw, The Yw axis component and the Zw axis component are shown.

A conversion formula between the coordinates (x, y, z) of the camera coordinate system X, Y, Z and the coordinates (xw, yw, zw) of the world coordinate system Xw, Yw, Zw is expressed by Formula 1.

Here, assuming that the focal length of the camera C_3 is “f”, the coordinates (xp, yp) of the coordinate system Xp • Yp of the imaging surface S and the coordinates (x, y, z) of the camera coordinate system X • Y • Z are The conversion formula between is expressed by Equation 2.

Also, Equation 3 is obtained based on Equation 1 and Equation 2. Equation 3 shows a conversion formula between the coordinates (xp, yp) of the coordinate system Xp / Yp of the imaging surface S and the coordinates (xw, zw) of the two-dimensional road surface coordinate system Xw / Zw.

Also, a bird's eye coordinate system X3 / Y3 that is a coordinate system of the bird's eye image BEV_3 shown in FIG. 4C is defined. The bird's-eye coordinate system X3 / Y3 is a two-dimensional coordinate system having the X3 axis and the Y3 axis as coordinate axes. When the coordinates in the bird's-eye view coordinate system X3 / Y3 are expressed as (x3, y3), the position of each pixel forming the bird's-eye view image BEV_3 is represented by the coordinates (x3, y3). “X3” and “y3” respectively indicate an X3 axis component and a Y3 axis component in the bird's eye view coordinate system X3 · Y3.

The projection from the two-dimensional coordinate system Xw / Zw representing the road surface to the bird's eye coordinate system X3 / Y3 corresponds to a so-called parallel projection. When the height of the virtual camera, that is, the virtual viewpoint is “H”, the conversion formula between the coordinates (xw, zw) of the two-dimensional coordinate system Xw · Zw and the coordinates (x3, y3) of the bird's-eye coordinate system X3 · Y3 is , Represented by Equation (4). The height H of the virtual camera is determined in advance.

Further, based on Equation 4, Equation 5 is obtained, Equation 5 is obtained based on Equation 5 and Equation 3, and Equation 7 is obtained based on Equation 6. Equation 7 corresponds to a conversion formula for converting the coordinates (xp, yp) of the coordinate system Xp / Yp of the imaging surface S into the coordinates (x3, y3) of the bird's eye coordinate system X3 / Y3.

The coordinates (xp, yp) of the coordinate system Xp / Yp of the imaging surface S represent the coordinates of the object scene image P_3 captured by the camera C_3. Accordingly, the object scene image P_3 from the camera C3 is converted into the bird's-eye view image BEV_3 by using Equation 7. Actually, the object scene image P_3 is first subjected to image processing such as lens distortion correction, and then converted into a bird's-eye view image BEV_3 by Equation 7.

Referring to FIGS. 10 and 11, when the obstacle (three-dimensional object) 200 exists in the common left visual field VW_34 of the vehicle 100, the obstacle captured by the camera C_3 is displayed as the obstacle image OBJ_3 in the bird's-eye view image BEV_3. The obstacle that appears and is captured by the camera C_4 appears in the bird's-eye view image BEV_3 as an obstacle image OBJ_4.

The postures of the obstacle images OBJ_3 and OBJ_4 are different from each other due to the difference between the viewpoint of the camera C_3 and the viewpoint of the camera C_4. That is, the obstacle image OBJ_3 is reproduced so as to fall down along the connection line CL_3 connecting the camera C_3 and the bottom of the obstacle 200, and the obstacle image OBJ_4 is connected to the connection line CL_4 connecting the camera C_4 and the bottom of the obstacle 200. It is reproduced to fall down along.

When displaying such an obstacle 200 on the all-around bird's-eye view image, the CPU 12p executes the following processing.

As a premise of the processing, the reference line RF1a extends in parallel with the boundary line BL on the overlapping area OL_41 from the camera C_1 toward the common visual field VW_41. The reference line RF1b extends in parallel with the boundary line BL on the overlapping area OL_12 from the camera C_1 toward the common visual field VW_12. The reference line RF2a extends in parallel with the boundary line BL on the overlapping area OL_12 from the camera C_2 toward the common visual field VW_12. The reference line RF2b extends in parallel with the boundary line BL on the overlapping area OL_23 from the camera C_2 toward the common visual field VW_23.

The reference line RF3a extends in parallel with the boundary line BL on the overlapping area OL_23 from the camera C_3 toward the common visual field VW_23. The reference line RF3b extends in parallel with the boundary line BL on the overlapping area OL_34 from the camera C_3 toward the common visual field VW_34. The reference line RF4a extends in parallel with the boundary line BL on the overlapping area OL_34 from the camera C_4 toward the common visual field VW_34. The reference line RF4b extends in parallel with the boundary line BL on the overlapping area OL_41 from the camera C_4 toward the common visual field VW_41.

Further, the adjustment area BRG_12 is assigned to the overlapping area OL_12 in a manner that extends in a band shape to the front right of the vehicle 100, and the adjustment area BRG_23 is assigned to the overlapping area OL_23 in a manner that extends to the right rear of the vehicle 100. The adjustment area BRG_34 is assigned to the overlap area OL_34 in a manner that extends in a band shape to the left rear of the vehicle 100, and the adjustment area BRG_41 is assigned to the overlap area OL_41 in a manner that extends to the left front of the vehicle 100.

The edge in the width direction of the adjustment area BRG_12 extends in parallel with the reference lines RF1b and RF2a, and the edge in the width direction of the adjustment area BRG_23 extends in parallel with the reference lines RF2b and RF3a. The edge in the width direction of the adjustment area BRG_34 extends in parallel to the reference lines RF3b and RF4a, and the edge in the width direction of the adjustment area BRG_41 extends in parallel to the reference lines RF4b and RF1a.

In reproducing the obstacle 200 on the all-around bird's-eye view image, first, it is determined for each of the common visual fields VW_12 to VW_41 whether or not any obstacle exists. The obstacle 200 exists in the common visual field VW_34, and the corresponding obstacle images OBJ_3 and OBJ_4 are reproduced as the bird's-eye view images BEV_3 and BEV_4, respectively. Therefore, the obstacle image OBJ_3 is cut out from the bird's-eye view image BEV_3, and the obstacle image OBJ_4 is cut out from the bird's-eye view image BEV_4. Subsequently, it is determined what positional relationship the coordinates corresponding to the bottom of the obstacle 200 is with respect to the adjustment area BRG_34.

As shown in the upper part of FIG. 12, when the coordinates corresponding to the bottom of the obstacle 200 are present on the camera C_3 side with respect to the adjustment area BRG_34, the obstacle image OBJ_3 is determined as the selected image S_34 in the common visual field VW_34. Also, as shown in the upper part of FIG. 13, when the coordinates corresponding to the bottom of the obstacle 200 are present on the camera C_4 side with respect to the adjustment area BRG_34, the obstacle image OBJ_4 is determined as the selected image S_34 in the common visual field VW_34. .

Furthermore, as shown in FIG. 14, when coordinates corresponding to the bottom of the obstacle 200 are present on the adjustment area BRG_34, a connection line CL_4 connecting the camera C_4 and the bottom of the obstacle 200 is defined on the bird's-eye view image BEV_4. Is done. Further, the difference between the defined angle of the connecting line CL_4 and the angle of the boundary line BL assigned to the common visual field VW_34 (that is, the angle of the reference lines RF3b and RF4a) is calculated as “Δθ”.

The posture of the obstacle image OBJ_4 is corrected so as to be along the boundary line BL (that is, the reference lines RF3b and RF4a) with reference to the calculated difference Δθ. As a result of such correction processing, the direction in which the obstacle image OBJ_4 falls is coincident with the length direction of the boundary line BL (that is, the reference lines RF3b and RF4a) as shown in the lower part of FIG. The obstacle image OBJ_4 having the corrected posture is then determined as the selected image S_34 in the common visual field VW_34.

Note that when the coordinates corresponding to the bottom of the obstacle 200 transition within the adjustment area BRG_34 as shown in FIG. 15A, the posture of the obstacle image OBJ_4 is corrected as shown in FIG. 15B.

In creating the all-around bird's-eye view image, images outside the boundary line BL are deleted from each of the bird's-eye view images BEV_1 to BEV_4, and the bird's-eye view images BEV_1 to BEV_4 remaining after the deletion are combined with each other (FIG. 5 to FIG. 5). (See FIG. 6). The selected image S_34 is pasted on the combined image, thereby completing the all-around bird's-eye view image. The obstacle image OBJ_3 or OBJ_4 is reproduced in the overlap area OL_34 of the completed all-around bird's-eye view image as shown in the lower part of FIG. 12, the lower part of FIG. 13, or the lower part of FIG.

Specifically, the CPU 12p executes processing according to the flowcharts shown in FIGS. The control program corresponding to these flowcharts is stored in the flash memory 14 (see FIG. 1).

In step S1 shown in FIG. 16, the scene images P_1 to P_4 are captured from the cameras C_1 to C_4. In step S3, bird's-eye view images BEV_1 to BEV_4 are created based on the captured object scene images P_1 to P_4. The created bird's-eye view images BEV_1 to BEV_4 are secured in the work area W1. In step S5, an obstacle detection process is executed. In step S7, the all-around bird's-eye view image is created based on the bird's-eye view images BEV_1 to BEV_4 created in step S3. The created all-around bird's-eye view image is secured in the work area W2. On the monitor screen of the display device 16, a driving support image based on the all-around bird's-eye view image secured in the work area W2 is displayed. When the process of step S7 is completed, the process returns to step S1.

The obstacle detection process shown in step S5 follows the subroutine shown in FIGS. First, in step S11, variables M and N are set to “1” and “2”, respectively. Here, the variable M is a variable that is updated in the order of “1” → “2” → “3” → “4”, and the variable N is “2” → “3” → “4” → “1”. It is a variable that is updated in order.

In step S13, a difference image in the common visual field VW_MN is created based on the bird's-eye view image BEV_M based on the output of the camera C_M and the bird's-eye view image BEV_N based on the output of the camera C_N. In step S15, it is determined based on the difference image created in step S13 whether an obstacle exists in the common visual field VW_MN.

If no obstacle is detected from the common visual field VW_MN, the flag FLG_MN is set to “0” in step S17, and the process proceeds to step S23. On the other hand, if an obstacle is detected from the common visual field VW_MN, the flag FLG_MN is set to “1” in step S19, obstacle image processing is executed in step S21, and then the process proceeds to step S23.

In step S23, it is determined whether or not the variable M indicates “4”. If the variable M is less than “4”, the variables M and N are updated in step S25, and the process returns to step S13. On the other hand, if the variable M is “4”, the process returns to the upper layer routine.

The obstacle image processing in step S21 shown in FIG. 17 is executed according to a subroutine shown in FIGS. First, in step S31, the obstacle image OBJ_M is cut out from the bird's-eye image BEV_M, and the obstacle image OBJ_N is cut out from the bird's-eye image BEV_N.

In step S33, it is determined whether or not coordinates corresponding to the bottom of the obstacle are present on the camera C_M side with respect to the adjustment area BRG_MN. In step S35, it is determined whether or not the coordinates corresponding to the bottom of the obstacle exist on the camera C_N side with respect to the adjustment area BRG_MN.

If “YES” in the step S33, the process proceeds to a step S37 to determine the obstacle image OBJ_M as the selected image S_MN. If “NO” in the step S33, but if “YES” in the step S35, the process proceeds to a step S39 to determine the obstacle image OBJ_N as the selected image S_MN. When the process of step S37 or S39 is completed, the process returns to the upper hierarchy routine.

If both step S33 and S35 are NO, the process proceeds to step S41 and subsequent steps. In step S41, a connection line CL_N connecting the camera C_N and the bottom of the obstacle is defined on the bird's-eye view image BEV_N. In step S43, the difference in angle between the connecting line CL_N defined in step S41 and the boundary line BL_MN is calculated as “Δθ”. In step S45, the posture of the obstacle image OBJ_N is corrected with reference to the calculated difference Δθ. In step S47, the obstacle image OBJ_N having the corrected posture is determined as the selected image S_MN. When the process of step S47 is completed, the process returns to the upper-level routine.

The all-around bird's-eye view creation process in step S7 shown in FIG. 16 follows a subroutine shown in FIGS. First, in step S51, the variable M is set to “1”. As described above, the variable M is a variable that is updated in the order of “1” → “2” → “3” → “4”. In step S53, an image outside the boundary line is deleted from the bird's-eye view image BEV_M, and in step S55, it is determined whether or not the variable M has reached “4”. If the variable M is less than “4”, the variable M is updated in step S57, and then the process returns to step S53. On the other hand, if the variable M indicates “4”, the process proceeds to step S59, and the bird's-eye images BEV_1 to BEV_4 processed in step S53 are coupled to each other by coordinate transformation.

When the combined image is completed, variables M and N are set to “1” and “2”, respectively, in step S61. As described above, the variable M is a variable that is updated in the order of “1” → “2” → “3” → “4”, and the variable N is “2” → “3” → “4” → “1”. It is a variable that is updated in this order.

In step S63, it is determined whether or not the flag FLG_MN is “1”. If the determination result is NO, the process proceeds to step S67 as it is. If the determination result is YES, the selected image S_MN is created in step S59. Paste to image. In step S67, it is determined whether or not the variable M has reached “4”. If NO, the variables M and N are updated in step S69, and then the process returns to step S63. Return.

As can be seen from the above description, when attention is paid to the cameras C_3 and C_4 among the cameras C_1 to C_4, the cameras C_3 and C_4 partially have a common visual field VW_34 and are obliquely intersecting the road surface. Capture the scene. The CPU 12p captures the scene images P_3 and P_4 output from the cameras C_3 and C_4, respectively (S1), and creates the bird's-eye view image for the road surface based on the captured scene images P_3 and P_4 (S3, S7). . The CPU 12p also sets the posture of the obstacle image appearing in the bird's-eye view image along the reference lines RF3b and RF4a extending parallel to each other from the cameras C_3 and C_4 toward the object scene when the obstacle exists in the common visual field VW_34. Adjust (S15, S21).

Each of the cameras C_3 and C_4 captures an object scene in a direction that obliquely intersects the road surface. For this reason, when an obstacle exists on the road surface, the obstacle image appears in the bird's-eye view image in a posture of falling along a line connecting the camera C_3 or C_4 and the obstacle. Further, the posture of the obstacle image changes due to a change in the positional relationship between the three-dimensional object and the camera C_3 or C_4.

According to this embodiment, when an obstacle is detected from the common visual field VW_34, the posture of the obstacle image appearing in the bird's-eye view image is such that the reference lines RF3b and RF4a extend parallel to each other from the cameras C_3 and C_4 toward the object scene. It is adjusted to follow. Thereby, the change in the posture of the obstacle image due to the change in the positional relationship between the obstacle and the camera C_3 or C_4 is suppressed, and the visibility of the obstacle in the common visual field VW_34 is improved.

In this embodiment, when the bird's-eye view images BEV_1 to BEV_4 are combined, a part of the image outside the boundary line BL is deleted (see FIG. 5). However, you may make it synthesize | combine two partial images showing a common visual field with equal weighting amount.

In this embodiment, the reference lines RFMb and RFNa are extended in parallel to the boundary line BL on the overlapping area OL_MN (M: 1 → 2 → 3 → 4, N: 2 → 3 → 4 → 1). . However, the reference lines RFMb and RFNa need only be parallel to each other, and are not necessarily parallel to the boundary line BL on the overlapping area OL_MN. For example, the reference lines RF3b and RF4a may be defined as shown in FIG.

Further, when the reference lines RFMb and RFNa are extended so as not to be parallel to each other as shown in FIG. 23 or FIG. 24, the shortest distance between the obstacle and the boundary line BL is calculated and the posture adjustment as shown in FIG. 15 is performed. It is preferred to do so.

In this embodiment, when an obstacle is found from the common visual field VW_MN, obstacle image processing is executed regardless of the size of the obstacle (see FIG. 17). However, when the obstacle is huge like a house or a building, if the posture of the obstacle image is corrected, the reproducibility of the obstacle may be lowered.

Such a concern can be solved by adding the processing of steps S71 to S73 shown in FIG. Referring to FIG. 25, in step S71, the size of the obstacle is detected, and in step S73, it is determined whether or not the detected size is below a threshold value THsz. If the determination result is NO, the process proceeds to step S19, and if the determination result is YES, the process proceeds to step S21.

In this embodiment, an obstacle is detected from the common visual field VW_MN based on the difference image created for the common visual field VW_MN (see steps S13 to S15 in FIG. 17). However, when detecting an obstacle, a stereo vision method or an optical flow method may be used, and an ultrasonic sensor, a millimeter wave sensor, or a microwave sensor may be used.

Further, in this embodiment, when the coordinates corresponding to the bottom of the obstacle exists in the adjustment area BRG_MN, the posture of the obstacle image OBJ_N corresponding to the camera C_N is corrected, and the obstacle image OBJ_N having the corrected posture is corrected. Is determined as the selected image S_MN.

However, which of the obstacle images OBJ_M and OBJ_N is subjected to such processing depends on the difference in height between the cameras C_M and C_N, the difference in distance from each of the cameras C_M and C_N to the obstacle, the cameras C_M and You may make it determine based on the appearance of the obstruction from each of C_N. Further, the postures of both the obstacle images OBJ_M and OBJ_N are corrected so as to follow the reference lines RFMb and RFNa, the obstacle images OBJ_M and OBJ_N having the corrected postures are synthesized, and the synthesized obstacle image is selected as the selected image S_MN. You may make it determine as.

It should be noted that utility poles, signs, bicycles, motorcycles, vending machines, etc. are clearly different depending on the viewpoint. Therefore, if specific obstacles such as utility poles, signs, bicycles, motorcycles, vending machines, etc. are registered in advance, and the selected images are determined based on the visibility of registered obstacles, the reproduction of the obstacles A decline in sex can be avoided.

In this embodiment, the obstacle image cut out from the bird's-eye view image is pasted on the all-around bird's-eye view image (see steps S31, S37, S39 to S47 in FIG. 18, and step S65 in FIG. 21). However, the obstacle image that appears in the object scene image before being converted into the bird's-eye view image may be pasted on the all-round bird's-eye image.

The following are notes on the above examples. This annotation can be arbitrarily combined with the above-described embodiment as long as there is no contradiction.

The coordinate transformation for generating a bird's-eye view image from a photographed image as described in the embodiment is generally called perspective projection transformation. Instead of using the perspective projection conversion, a bird's eye view image may be generated from the captured image by a known plane projective conversion. When using planar projective transformation, a homography matrix (coordinate transformation matrix) for converting the coordinate value of each pixel on the captured image into the coordinate value of each pixel on the bird's eye view image is obtained in advance at the stage of camera calibration processing. . A method for obtaining a homography matrix is known. And when performing image conversion, what is necessary is just to convert a picked-up image into a bird's-eye view image based on a homography matrix. In any case, the captured image is converted into the bird's-eye view image by projecting the captured image onto the bird's-eye view image.

Although the present invention has been described and illustrated in detail, it is clear that it has been used merely as an illustration and example and should not be construed as limiting, and the spirit and scope of the present invention are attached Limited only by the wording of the claims.

DESCRIPTION OF SYMBOLS 10 ... Steering assistance apparatus C1-C4 ... Camera 12 ... Image processing circuit 12p ... CPU
12m ... Memory 14 ... Flash memory 16 ... Display device 100 ... Vehicle 200 ... Obstacle

Claims

An image processing device comprising:
A plurality of cameras, each having a common field of view and capturing a scene in a direction that obliquely intersects the reference plane;
Creating means for creating a bird's-eye view image with respect to the reference plane based on a plurality of object scene images respectively output from the plurality of cameras;
Discrimination means for discriminating a positional relationship between the three-dimensional object existing in the common visual field and the plurality of cameras; and a posture of the three-dimensional object image on the bird's-eye image created by the creation means based on a discrimination result of the discrimination means. Adjustment means to adjust.
The image processing apparatus according to claim 1, wherein the determination unit determines whether or not the three-dimensional object is sandwiched by a plurality of reference lines extending in a predetermined manner from the plurality of cameras toward the object scene,
The adjusting means adjusts the posture of the three-dimensional object image along the plurality of reference lines.
The image processing apparatus according to claim 2, wherein the predetermined mode corresponds to a mode in which the plurality of reference lines are parallel to each other.
The image processing apparatus according to claim 2, wherein the three-dimensional object image corresponds to a bird's-eye image of the three-dimensional object captured by a reference camera that is one of the plurality of cameras.
The adjusting means includes a defining means for defining a line connecting the reference camera and the solid object when the solid object belongs to a field of view sandwiched by the plurality of reference lines, and the line defined by the definition means and the reference Correction means for correcting the posture of the three-dimensional object image with reference to the difference in angle with the line is included.
The image processing apparatus according to claim 4, wherein the adjustment unit selects, as the reference camera, a camera closer to the three-dimensional object among the plurality of cameras when the three-dimensional object deviates from a field of view sandwiched by the plurality of reference lines. Selection means to be included.
The image processing apparatus according to claim 1, further comprising a restriction unit that restricts the adjustment process of the adjustment unit when the three-dimensional object meets a specific condition.