JP2013005032A - On-vehicle camera posture detection device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an on-vehicle camera posture detection device capable of accurately detecting the posture of a camera.SOLUTION: An on-vehicle camera posture detection device includes: a pre-movement image storage section 20 for storing a pre-movement image of a calibration sheet 200 imaged by a camera 10; a pre-movement camera posture estimation section 24 for estimating the posture of the camera 10 on the basis of the pre-movement image; a post-movement image storage section 22 for storing a post-movement image imaged after moving a vehicle linearly by a prescribed distance; a post-movement camera posture estimation section 26 for estimating the posture of the camera 10 on the basis of the post-movement image; a deviation amount calculation section 32 for calculating a deviation amount of the calibration sheet on the basis of the two estimation results; a camera posture determination section 34 for determining the posture of the camera 10 on the basis of the camera 10 posture estimation value and the deviation amount of the calibration sheet; and a linear movement determination section 46 for determining whether the movement of the vehicle is linear or not.

Description

  The present invention relates to an in-vehicle camera attitude detection device that detects an attachment angle or the like of a camera that is mounted on a vehicle and captures a peripheral image.

  When accurately detecting the position and size of the subject included in this image based on the image taken by the camera mounted on the vehicle, or by shooting adjacent areas with multiple cameras, In the case of continuous composition, it is necessary to know the exact installation position and angle of the camera.

  In general, the installation position and angle of a camera are often determined in advance when designing the entire system including the camera, but in practice, since installation errors are included, calibration is performed after installation to ensure the correct installation position and angle. An angle is detected. For example, conventionally, a calibration sheet on which a calibration pattern including a plurality of calibration indices is printed is arranged at a predetermined position near the vehicle, and an image obtained as a result of photographing the calibration sheet with a camera is used. A camera attachment angle calculation method that detects the installation position and angle of a camera is known (for example, see Patent Document 1).

JP 2010-183265 A (page 2-11, FIG. 1-23)

  By the way, in the conventional method described above, it is assumed that the calibration sheet is accurately arranged at a predetermined position. If there is a deviation in the installation position of the calibration sheet, the posture (position and angle) of the camera is accurately determined. There was a problem that it could not be detected. For example, it is assumed that the calibration sheet is arranged behind the vehicle so that the installation position is in contact with the end of the rear bumper, and this is photographed with a camera installed behind the vehicle. Even when the position of the calibration sheet is designated in this way, the rear bumper is located away from the ground, and it is difficult to align the position of the calibration sheet with the end portion. The same applies to the left and right positions of calibration, and it is not easy to accurately align the center of the vehicle and the center of the calibration sheet.

  The present invention has been created in view of the above points, and an object thereof is to provide an in-vehicle camera posture detection device capable of accurately detecting the posture of a camera.

  In order to solve the above-described problem, an on-vehicle camera posture detection device of the present invention photographs a plurality of calibration indices included in a calibration sheet arranged around a vehicle with a camera mounted on the vehicle, and obtained images Is a vehicle-mounted camera posture detection device that detects the posture of a camera based on the pre-movement image storage unit that stores an image of a calibration sheet photographed by the camera as a pre-movement image, and is stored in the pre-movement image storage unit. A first posture estimating means for estimating the posture of the camera based on the pre-movement image, and a movement for storing an image of a calibration sheet photographed by the camera after moving the vehicle in a straight line as a post-movement image Post-image storage means; second posture estimation means for estimating the posture of the camera based on the post-movement image stored in the post-movement image storage means; Posture estimated by at least one of sheet deviation amount calculating means for calculating the deviation amount of the calibration sheet from a predetermined position based on the estimation result by the second posture estimating means, and first and second camera posture estimating means. Based on the estimated value and the amount of deviation calculated by the sheet deviation amount calculation means, camera posture determination means for determining the posture of the camera, and the vehicle from the acquisition of the pre-movement image to the acquisition of the post-movement image Linear movement determination means for determining whether or not the movement has been made linear.

  Since the amount of deviation of the calibration sheet from the specified position (design position) is calculated based on the images before and after the vehicle is moved, the effects of the deviation of the calibration sheet can be removed, and the camera posture can be accurately determined. Can be detected. In addition, by making the movement of the vehicle linear, when the calibration sheet is photographed before and after the movement, the deviation in the rotation direction of the calibration sheet can be made constant, and the number of parameters to be calculated is reduced. Therefore, the process can be simplified. Furthermore, since it is determined whether or not the movement of the vehicle is a straight line, the movement of the vehicle can be redone if it is not a straight line, thereby preventing an increase in errors included in the detection of the camera posture. be able to.

  In addition, a plurality of images obtained by photographing with the camera at a plurality of positions corresponding to the front and rear and middle positions of the movement of the vehicle are obtained by photographing through the lens by the camera described above. And a straight line movement determining means for determining whether or not the vehicle has been moved linearly based on a plurality of images after the distortion has been removed by the distortion correcting means. It is desirable to judge. Specifically, the above-described linear movement determination unit extracts feature points corresponding to the same part of the calibration sheet for a plurality of images after distortion removal, and each of the plurality of feature points included in the plurality of images. It is desirable to determine that the movement of the vehicle has been made linear. By using distortion corrected images to remove distortion, the movement of the subject (feature point) in the image linked to the movement of the vehicle becomes linear. Judgment becomes easy.

  In addition, it is desirable to further include a display processing unit that displays an image on which distortion correction has been performed by the above-described distortion correction unit together with a plurality of feature points on a display device. In particular, it is desirable that the image displayed on the display device by the display processing means described above includes a linear auxiliary line that passes through both ends of the plurality of feature points. As a result, the driver who moves the vehicle can easily check whether the movement of the vehicle is linear while viewing the image.

  In addition, it is desirable that the above-described linear movement determination means notifies the vehicle driver when it is determined that the vehicle has not been moved linearly. Accordingly, the driver of the vehicle can be surely notified that the movement of the vehicle is not linear, and a series of operations relating to the determination of the camera posture can be urged to be performed again.

It is a figure which shows the structure of the vehicle-mounted camera attitude | position detection apparatus of one Embodiment. It is a figure which shows the specific example of installation of a camera. It is a figure which shows the specific example of a calibration sheet | seat. It is a flowchart which shows the rough operation | movement procedure which determines the attitude | position of a camera. It is a figure which shows the relationship of three deviation | shift amounts. It is a figure which shows the detailed structure of the camera attitude | position estimation part before a movement. It is explanatory drawing of coordinate transformation. It is a flowchart which shows the operation | movement procedure of the straight line determination performed in parallel with the movement of a vehicle. It is a figure which shows the example of a display of the image before and behind distortion correction.

  Hereinafter, an in-vehicle camera attitude detection device according to an embodiment to which the present invention is applied will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration of an in-vehicle camera attitude detection device according to an embodiment. As shown in FIG. 1, the on-vehicle camera posture detection device 100 of this embodiment includes a camera 10, an image capture unit 12, an image memory 14, a pre-movement image storage unit 20, an intermediate image storage unit 21, and a post-movement image storage unit. 22, pre-movement camera posture estimation unit 24, post-movement camera posture estimation unit 26, movement distance calculation unit 30, displacement amount calculation unit 32, camera posture determination unit 34, display device 40, display processing unit 42, distortion correction unit 44, The linear movement determination unit 46 and the operation unit 50 are included. The camera posture detected by the vehicle-mounted camera posture detection device 100 is for using the detection result in another device using the camera, and the configuration of the vehicle-mounted camera posture detection device 100 is included in the other device. You may make it let.

  The camera 10 is attached to a predetermined position at the rear of the vehicle at a predetermined angle, and images the rear of the vehicle through a fisheye lens or a wide-angle lens. By photographing a subject using a fisheye lens or the like, an image having a distorted periphery with respect to the center can be obtained. FIG. 2 is a diagram illustrating a specific installation example of the camera 10. In the example shown in FIG. 2, the camera 10 is attached to the center of the rear bumper at an angle of 30 ° from the horizontal to the downward. In the present embodiment, the calibration sheet 200 is arranged on the ground behind the vehicle so as to be included in the shooting range of the camera 10.

  FIG. 3 is a diagram illustrating a specific example of the calibration sheet 200. A calibration sheet 200 shown in FIG. 3 includes twelve small black circles 210 and six large white circles 220 as calibration indices (calibration marks). The large white circles 220 are arranged in two rows as seen from the vehicle side so that three are included in each row. Further, each of the three large white circles 220 constituting the column far from the vehicle includes four small black circles 210 arranged in two rows and two columns. These small black circles 210 and large white circles 220 are identified on the basis of an image captured by photographing, and the respective center (circular center) coordinates are obtained and used for various calculations.

  The image capturing unit 12 captures an image obtained by photographing the rear of the vehicle with the camera 10. The captured image is temporarily stored in the image memory 14. In the present embodiment, the camera 10 performs the first shooting before moving the vehicle with the calibration sheet 200 placed at a predetermined position behind the vehicle, and then advances the vehicle by a predetermined distance and then moves the vehicle with the camera 10. Take a second shot after moving.

  The pre-movement image storage unit 20 reads out an image captured before the movement of the vehicle and stored in the image memory 14 and stores it as a pre-movement image. The intermediate image storage unit 21 reads out an image captured during the movement of the vehicle and stored in the image memory 14 and stores it as an intermediate image. Further, the post-movement image storage unit 22 reads out an image captured after the vehicle has moved and stored in the image memory 14 and stores it as a post-movement image.

  The pre-movement camera posture estimation unit 24 estimates the posture of the camera 10 based on the position of the calibration index in the calibration sheet 200 based on the pre-movement image stored in the pre-movement image storage unit 20. Camera positions x1, y1, z1 and camera angles rv1, rh1, rr1 are obtained as estimated posture values. Here, x1 is an X coordinate value, y1 is a Y coordinate value, z1 is a Z coordinate value, rv1 is a pitch angle, rh1 is a yaw angle, and rr1 is a roll angle. In addition, an X direction and a Y direction orthogonal to each other are set along a plane parallel to the ground on which the calibration sheet 200 is arranged, and a Z direction is set in a direction perpendicular to the ground.

  The post-movement camera posture estimation unit 26 estimates the posture of the camera 10 based on the position of the calibration index in the calibration sheet 200 based on the post-movement image stored in the post-movement image storage unit 22. Camera positions x2, y2, z2 and camera angles rv2, rh2, rr2 are obtained as estimated posture values. The meaning of x2 etc. is the same as the meaning of x1 etc.

  The movement distance calculation unit 30 is based on the estimated posture value obtained by the pre-movement camera posture estimation unit 24 and the estimated posture value obtained by the post-movement camera posture estimation unit 26, and is the distance that the vehicle has actually moved. d is calculated.

  Based on the estimated posture value obtained by the pre-movement camera posture estimator 24 and the estimated posture value obtained by the post-movement camera posture estimator 26, the deviation amount calculation unit 32 determines the normal position of the calibration sheet 200. The amount of deviation from (ideal design position assumed at the time of product design) is calculated.

  Based on the estimated value obtained by at least one of the pre-movement camera posture estimation unit 24 and the post-movement camera posture estimation unit 26 and the deviation amount calculated by the deviation amount calculation unit 32, the camera posture determination unit 34 10 postures are determined.

  The display processing unit 42 displays an image captured by the camera 10 on the display device 40 when the vehicle is moved. The distortion correction unit 44 includes a plurality of images (pre-movement image storage unit 20, intermediate image storage unit 21, post-movement image storage unit) obtained by photographing with the camera 10 at a plurality of locations corresponding to before and after movement of the vehicle and intermediate positions. 22), distortion correction is performed on the pre-movement image, the intermediate image, and the post-movement image stored in the image. On the image after distortion correction, the trajectory of the vehicle that has moved linearly becomes a straight line. The linear movement determination unit 46 determines whether or not the vehicle has been moved linearly based on the plurality of images after the distortion is removed by the distortion correction unit 44. Specifically, the straight line movement determination unit 46 extracts feature points corresponding to the same portion of the calibration sheet 200 from the plurality of images after distortion removal, and each of the plurality of feature points included in the plurality of images. Are arranged in a straight line, it is determined that the movement of the vehicle has been made in a straight line.

  The operation unit 50 is for receiving a user's operation instruction, and includes various keys. For example, the image capturing unit 12 captures images before and after movement and at an intermediate position in response to a capture instruction performed by the user operating the operation unit 50.

  The above-mentioned pre-movement image storage unit 20 is the pre-movement image storage unit, the post-movement image storage unit 22 is the post-movement image storage unit, the pre-movement camera posture estimation unit 24 is the first posture estimation unit, and the post-movement camera posture. The estimation unit 26 is a second posture estimation unit, the deviation amount calculation unit 32 is a sheet deviation amount calculation unit, the camera posture determination unit 34 is a camera posture determination unit, the display processing unit 42 is a display processing unit, and a distortion correction unit. Reference numeral 44 corresponds to the distortion correction unit, and the linear movement determination unit 46 corresponds to the linear movement determination unit.

  The in-vehicle camera posture detection apparatus 100 according to the present embodiment has such a configuration. Next, an operation from when the camera 10 captures the calibration sheet 200 and finally determines the posture of the camera 10 will be described. To do.

  FIG. 4 is a flowchart showing a schematic operation procedure for determining the posture of the camera 10. First, the first shooting by the camera 10 is performed in a state where the calibration sheet 200 is disposed at a predetermined position on the rear side of the vehicle (step 100). For example, the user operates the operation unit 50 to give a first capture instruction. The image capture unit 12 captures an image captured by the camera 10 in response to the capture instruction and stores it in the image memory 14. The pre-movement image storage unit 20 reads out an image from the image memory 14 and stores it as a pre-movement image.

  Further, the pre-movement camera posture estimation unit 24 reads the pre-movement image from the pre-movement image storage unit 20 and estimates the posture of the camera 10 (step 102). Thereby, camera positions x1, y1, z1 and camera angles rv1, rh1, rr1 are obtained as estimated values of the posture of the camera 10.

  Next, the vehicle is moved by a predetermined distance (step 104). This movement of the vehicle is performed by advancing the vehicle in a straight line by a predetermined distance. There is an assumed ideal value (for example, 20 cm) as the predetermined distance, but there is no particular problem even if it is slightly deviated from the ideal value in the vicinity thereof. In the present embodiment, it is determined whether or not the movement of the vehicle has been made linear (straight line determination). If the vehicle has not been made linear, a predetermined notification is given, but these operations will be described later. .

  Next, the second shooting is performed by the camera 10 (step 106). For example, the user operates the operation unit 50 to give a second capture instruction. The image capture unit 12 captures an image captured by the camera 10 in response to the capture instruction and stores it in the image memory 14. The post-movement image storage unit 22 reads an image from the image memory 14 and stores it as a post-movement image.

  The post-movement camera posture estimation unit 26 reads the post-movement image from the post-movement image storage unit 22 and estimates the posture of the camera 10 (step 108). Thereby, camera positions x2, y2, z2 and camera angles rv2, rh2, rr2 are obtained as estimated values of the posture of the camera 10.

  Next, the moving distance calculation unit 30 calculates the moving distance d of the vehicle based on the two camera positions obtained by the estimation operations in steps 102 and 108 (step 110). Specifically, the moving distance calculation unit 30 calculates the distance d by the following expression using the camera positions x1 and y1 that are the first estimation results and the camera positions x2 and y2 that are the second estimation results. .

d = √ ((x1-x2) ² + (y1-y2) ² ) (1)
Next, the deviation amount calculation unit 32 calculates deviation amounts X, Y, and R from the design position of the calibration sheet 200 (step 112). Here, X is the amount of deviation in the X direction, Y is the amount of deviation in the Y direction, and R is the amount of deviation in the rotational direction. FIG. 5 is a diagram showing the relationship between the three deviation amounts X, Y, and R. In FIG. 5, A shows one point in the calibration sheet 200 arranged at the design position, and B shows a corresponding point in the calibration sheet 200 actually arranged.

  Assume that the position of the camera 10 obtained when the calibration sheet 200 is arranged at the design position before the vehicle is moved (this position is referred to as “design position before movement”) is x and y. Since the position in the z direction is constant regardless of the displacement of the calibration sheet 200, description thereof is omitted. Between the position x1, y1 of the camera 10 estimated before the vehicle movement and the design position x, y before the movement,

There is a relationship.

  Similarly, the position of the camera 10 obtained when it is assumed that the calibration sheet 200 is arranged at the design position after moving the vehicle (this position is referred to as “designed position after movement”) is x ′, y ′. And Between the positions x2, y2 of the camera 10 estimated after the vehicle movement and the designed positions x ', y' after the movement,

There is a relationship. When formulas (2) and (3) are expanded,

It becomes. When the expression (4) is expressed by a determinant focusing on the shift amount (X, Y, R) of the calibration sheet 200,

It becomes. When this equation (5) is modified, the following equation (6) is obtained.

By the way, if the vehicle is moved along the Y direction, the following relationship exists between the design positions x, y before the camera 10 moves and the design positions x ', y' after the movement.

x ′ = x
y ′ = y + d (7)
That is, all the right sides of the equation (6) are known values, and the deviation amount calculation unit 32 can calculate the deviation amounts X, Y, and R by using the equation (6).

  Next, the camera posture determination unit 34 calculates angles rv (pitch angle), rh (yaw angle), and rr (roll angle) of the camera 10, which are unknown values (step 114). The relationship between the actual angles rv, rh, rr of the camera 10, the estimated values rv1, rh1, rr1, and the deviation amount R of the calibration sheet 200 can be expressed as follows.

Ry (R), Rx (rv), Ry (rh), Rz (rr)
= Rx (rv1) .Ry (rr1) .Rz (rh1) (8)
Here, Rx (rv), Ry (rh), and Rz (rr) are rotation matrices shown below.

When the equation (8) is modified, it becomes as follows.

Rx (rv), Ry (rh), Rz (rr)
= Ry (-R) .Rx (rv1) .Ry (rh1) .Rz (rr1) (12)
Since all the right sides of the equation (12) are known values, they are expanded and placed as follows.

Further, when the left side of the expression (12) is expanded and solved together with the expression (13), the result is as follows.

In this way, the camera posture determination unit 34 calculates the angles rv, rh, and rr of the camera 10.

  Next, camera posture estimation operations by the pre-movement camera posture estimation unit 24 and the post-movement camera posture estimation unit 26 will be described. The pre-movement camera posture estimation unit 24 and the post-movement camera posture estimation unit 26 have the same configuration, and the details of the pre-movement camera posture estimation unit 24 will be described below.

  FIG. 6 is a diagram illustrating a detailed configuration of the pre-movement camera posture estimation unit 24. As shown in FIG. 6, the pre-movement camera posture estimation unit 24 includes a calibration index center position detection unit 60, calibration index coordinate storage units 61 and 63, a coordinate conversion unit 62, a configuration index distance adjustment unit 64, and a configuration index position adjustment. Part 65 and a camera posture adjusting part 66.

  The calibration index center position detection unit 60 reads the pre-movement image stored in the pre-movement image storage unit 20 and detects the center position of the calibration index in the calibration sheet 200 included in the pre-movement image. For example, the calibration index center position detection unit 60 extracts twelve small black circles 210 as calibration indices included in the calibration sheet 200 by image recognition, and stores the coordinate values of the respective center positions in the calibration index coordinate storage unit 61. save. The coordinate value of the center position is expressed by a fish-eye coordinate system in which an image plane (U-V coordinate system) is set perpendicular to the optical axis S of the camera 10.

  The coordinate conversion unit 62 converts the center coordinate value of each calibration index from the fish-eye coordinate system to a reference coordinate system in which the ground is the XY plane and the direction perpendicular to the ground is the Z direction, and the calibration index coordinate storage unit Save to 63.

  FIG. 7 is an explanatory diagram of coordinate conversion. In FIG. 7, XYZ represents a reference coordinate system, and UVS represents a fish-eye coordinate system. In FIG. 7, x1, y1, and z1 are distances between the origins of the fish-eye coordinate system UVS and the reference coordinate system XYZ, and rx, ry, and rz are reference coordinate systems XYZ. Assuming that the amount of rotation about each axis of the camera 10 is as follows, the following equation is established. However, u, v, and s are known as center coordinate values of the calibration index in the fish-eye coordinate system, and λ is a coefficient for converting the unit of the reference coordinate system from the actual distance between calibration indices to mm.

7B, when the optical axis of the camera 10 is directed to the point P on the XY plane, the rotation amount around the X axis is rv (= rx), and the rotation amount around the Y axis is rh (= ry), the amount of rotation about the Z axis is rr (= rz). Each of these is a pitch angle, a yaw angle, and a roll angle.

  Thereafter, the calibration index distance adjusting unit 64, the calibration index position adjusting unit 65, and the camera posture adjusting unit 66 are configured so that the axial distances between the calibration indexes become equal to the actual axial distances, and the calibration index positions. Rv, rh, rr, x1, y1, z1 are determined so that becomes equal to the actual calibration index position. That is, first, the calibration index distance adjusting unit 64 and the camera posture adjusting unit 66 change the rv, rh, and rr while fixing x1, y1, and z1, and set the center coordinate value between the calibration indexes as (15 ) To calculate the respective axial distances between the calibration indices, find rv, rh, rr in which the axial distances between the calibration indices are equal, and determine these rv, rh, rr as camera angles. Output as. Further, the calibration index position adjustment unit 65 and the camera posture adjustment unit 66 calculate the center coordinate value of each calibration index from Equation (15), and x1, y1, z1 that are equal to the actual center position (known) of the calibration index. Is output as the camera position.

  Specifically, the inclination of the calibration index can be adjusted by changing rr, the vertical distance between the calibration indices can be adjusted by changing rh, and the calibration can be performed by changing rv. The horizontal distance between the indicators can be adjusted.

  Therefore, first, the calibration index tilt adjustment amount determination unit 64a calculates the calibration index tilt angle using the center coordinate value of each calibration index when x1, y1, and z1 are constant values (for example, 0). When the alignment of the calibration indices is inclined, rr is adjusted so that the inclination angle = 0, and the adjustment amount Δrr is input to the camera attitude adjustment unit 66. The camera posture adjustment unit 66 calculates a new rr by rr = rr + Δrr and inputs it to the coordinate conversion unit 62. As a result, the coordinate conversion unit 62 calculates the center coordinates of each calibration index according to equation (15), and the calibration index tilt adjustment amount determination unit 64a uses the center coordinate value of each calibration index to determine the calibration index tilt angle. Calculate Thereafter, the above processing is continued until the calibration index inclination angle becomes 0, and rr is adjusted.

  When the adjustment of rr is completed so that the calibration index inclination angle becomes 0, the calibration index distance calculation unit 64b calculates the distance between the calibration indexes in the X-axis direction, and the calibration index distance adjustment completion determination unit 64d. Checks whether the distances in the X-axis direction between the calibration indices are equal, and if not, inputs an adjustment instruction signal to the calibration index distance adjustment amount determination unit 64c. Thus, the calibration index distance adjustment amount determination unit 64 c adjusts rh and inputs the adjustment amount Δrh to the camera posture adjustment unit 66. The camera posture adjustment unit 66 calculates a new rh from rh = rh + Δrh and inputs the new rh to the coordinate conversion unit 62. Thereby, the coordinate conversion unit 62 calculates the center coordinate value of each calibration index according to the equation (15), and the calibration index distance calculation unit 64b uses the center coordinate value of each calibration index to set the X axis between the calibration indices. The direction distance is calculated, and rh is adjusted until the X-axis direction distances between the calibration indices become equal.

  When the adjustment of rh is completed, the calibration index distance calculation unit 64b calculates the distance between the calibration indexes in the Y-axis direction, and the calibration index distance adjustment completion determination unit 64d determines the Y-axis direction distance between the calibration indexes. If they are not equal, an adjustment instruction signal is input to the calibration index distance adjustment amount determination unit 64c. As a result, the calibration index distance adjustment amount determination unit 64 c adjusts rv and inputs the adjustment amount Δrv to the camera posture adjustment unit 66. The camera attitude adjustment unit 66 calculates a new rv by rv = rv + Δrv and inputs it to the coordinate conversion unit 62. Thereby, the coordinate conversion unit 62 calculates the center coordinate value of each calibration index according to the equation (15), and the calibration index distance calculation unit 64b uses the center coordinate value of each calibration index to perform the Y-axis direction between the calibration indices. The distance is calculated, and rv is adjusted until the distance in the Y-axis direction between the calibration indices becomes equal.

  As described above, when adjustment of rv (= rv1), rh (= rh1), and rr (= rr1) is completed, the calibration index distance adjustment completion determination unit 64d outputs the attachment angle adjustment completion signal DAJED.

  When the attachment angle adjustment completion signal DAJED is generated, the calibration index position adjustment unit 65 obtains x1 and y1 that are equal to the actual calibration index center position (Xs, Ys) from the equation (15), and also calculates the calibration index size ( Alternatively, z1 is adjusted so that the distance between calibration indices is equal to the actual calibration index size (or distance between calibration indices), and is output as the camera position. Thereafter, the camera posture adjustment unit 66 outputs the calculated camera angles rv1, rh1, rr1 and camera positions x1, y1, z1 from the pre-movement camera posture estimation unit 24.

  As described above, the in-vehicle camera posture detection device 100 according to the present embodiment calculates the deviation amount from the predetermined position (design position) of the calibration sheet 200 based on the images before and after the vehicle is moved. It is possible to remove the influence due to the displacement of the motion sheet 200, and to accurately detect the posture of the camera 10. Further, by making the movement of the vehicle linear, when the calibration sheet 200 is photographed before and after the movement, the deviation in the rotation direction of the calibration sheet 200 can be made constant, and the number of parameters to be calculated is Since it can be reduced, processing can be simplified.

  Further, since the predetermined distance d for moving the vehicle is calculated based on the estimated values obtained by the pre-movement camera posture estimation unit 24 and the post-movement camera posture estimation unit 26, the vehicle movement distance d is accurately grasped. It is possible to reduce errors in camera posture estimation based on the post-movement image.

  Next, straight line determination performed when the vehicle is moved and notification of the result will be described. FIG. 8 is a flowchart showing an operation procedure of straight line determination performed in parallel with the movement of the vehicle. After the first shooting is completed and the pre-movement image is stored in the pre-movement image storage unit 20 (Step 100 in FIG. 4), the distortion correction unit 44 performs distortion correction on the pre-movement image (Step S100). 200). For example, this distortion correction is performed by projecting and converting the pre-movement image on a plane perpendicular to the optical axis of the camera 10. The image after distortion correction is stored in an image storage unit (not shown) (the same applies to an image after distortion correction corresponding to an intermediate image and a moved image described later).

  Next, an intermediate image is captured and stored in the intermediate image storage unit 21 (step 202). These operations are performed in the same procedure as that for capturing a pre-movement image. That is, after the vehicle is temporarily stopped, the user operates the operation unit 50 to instruct to capture an intermediate image. The image capture unit 12 captures an image captured by the camera 10 in response to the capture instruction and stores it in the image memory 14. The intermediate image storage unit 21 reads an image from the image memory 14 and stores it as an intermediate image. Further, the distortion correction unit 44 performs distortion correction on the intermediate image (step 204).

  In addition, after the second shooting is completed and the moved image is stored in the moved image storage unit 22 (step 106 in FIG. 4), the distortion correction unit 44 performs distortion correction on the moved image. (Step 206).

  Next, the straight line movement determination unit 46 extracts corresponding portions of the calibration sheet 200 included in a plurality of images after distortion correction corresponding to the pre-movement image, the intermediate image, and the post-movement image as feature points (step). 208). For example, the straight line movement determination unit 46 extracts three centers (circle centers) in the small black circles 210 as twelve calibration indexes included in the calibration sheet 200 shown in FIG. 3 as feature points. Further, the display processing unit 42 displays an image after distortion correction including the extracted feature points on the screen of the display device 40 (step 210).

  FIG. 9 is a diagram illustrating a display example of images before and after distortion correction. FIG. 9A shows an image before movement and an image after distortion correction. FIG. 9B shows an intermediate image and an image after distortion correction. FIG. 9C shows an image after movement and an image after distortion correction. The image after distortion correction displayed in step 210 may correspond to any of the pre-movement image, the intermediate image, and the post-movement image. However, in consideration of not disturbing the display of the extracted feature points, the post-movement image The one corresponding to is most desirable. This is because a road surface space is generated between the vehicle and the calibration sheet 200 as the vehicle moves. For example, in this embodiment, it is assumed that an image after distortion correction shown in FIG. 9C is displayed. This display image includes a marker M1 indicating the center of the three calibration indices of the calibration sheet 200 included in the pre-movement image, and a marker M2 indicating the center of the three calibration indices of the calibration sheet 200 included in the intermediate image. In addition, a marker M3 indicating the center of the three calibration indices of the calibration sheet 200 included in the post-movement image and an auxiliary line H connecting the markers M1 and M3 corresponding to the same position are included. Each marker M1, M2, M3 indicates the position of the feature point. When the movement of the vehicle is linear, these three markers M1, M2, M3 are arranged linearly. On the other hand, when the movement of the vehicle is not linear, these three markers M1, M2, and M3 are not arranged linearly, and the position of the center marker M2 deviates from the auxiliary line H. Therefore, the user (vehicle driver) can know by himself whether or not the movement of the vehicle is linear by looking at the display image. However, it may not be clear how far the marker M2 has deviated from the auxiliary line H and the vehicle has not moved linearly. In this embodiment, this determination is performed by the linear movement determination unit 46. ing.

  The linear movement determination unit 46 determines whether or not the vehicle has moved linearly (step 212). The determination method itself is basically the same as that performed by the user himself / herself. When the distance between the auxiliary line H and the marker M2 is equal to or smaller than a predetermined value, it is determined that the movement of the vehicle is linear, and the reference value is set. When exceeding, it determines with it not being linear. If it is not linear, a negative determination is made in the determination in step 212, and then the linear movement determination unit 46 notifies that the movement of the vehicle is not linear (step 214). For example, the straight line movement determination unit 46 displays a message prompting the user to redo the movement of the vehicle via the display processing unit 42. Alternatively, a buzzer sound or a message sound may be output from a buzzer or a speaker. The driver of the vehicle that has received this notification performs the movement of the vehicle again. In this case, since the vehicle cannot be accurately returned to the position where the pre-movement image was taken, the operation procedure shown in FIG. 4 is performed from the beginning.

  Thus, in the vehicle-mounted camera posture detection device 100 of the present embodiment, since it is determined whether or not the movement of the vehicle is linear, the movement of the vehicle can be redone if it is not linear, It is possible to prevent an increase in error included in the detection of the camera posture. In addition, since the movement of the subject (feature point) in the image interlocked with the movement of the vehicle is linear by using an image obtained by performing distortion correction and removing the distortion for the determination, the movement of the vehicle is linear. This makes it easy to determine whether or not

  Further, the vehicle is moved by displaying a distortion-corrected image including markers M1 to M3 as feature points corresponding to before and after the movement of the vehicle and an intermediate position and a linear auxiliary line H passing through both ends thereof. The driver himself / herself can easily confirm whether or not the movement of the vehicle is linear while viewing the image. Further, by notifying the vehicle driver when it is determined that the vehicle has not been moved linearly, it is possible to prompt the user to redo a series of operations related to determination of the camera posture.

  In addition, this invention is not limited to the said embodiment, A various deformation | transformation implementation is possible within the range of the summary of this invention. For example, in the above-described embodiment, the pre-movement image storage unit 20 and the pre-movement camera posture estimation unit 24, and the post-movement image storage unit 22 and the post-movement camera posture estimation unit 26 are provided separately. Since the same processing is performed, one may be omitted and the same processing may be repeated in the other.

  In the above-described embodiment, the camera 10 that captures the rear of the vehicle has been described. However, the present invention can also be applied when determining the posture of the camera that captures the front or side of the vehicle.

  As described above, according to the present invention, since the amount of deviation from the predetermined position (design position) of the calibration sheet 200 is calculated based on the images before and after the vehicle is moved, the deviation of the calibration sheet 200 is calculated. Thus, the posture of the camera 10 can be accurately detected. In addition, since it is determined whether or not the movement of the vehicle is a straight line, the movement of the vehicle can be redone if it is not a straight line, thereby preventing an increase in errors included in the detection of the camera posture. be able to.

DESCRIPTION OF SYMBOLS 10 Camera 12 Image capture part 14 Image memory 20 Pre-movement image storage part 21 Intermediate image storage part 22 Post-movement image storage part 24 Pre-movement camera attitude estimation part 26 Post-movement camera attitude estimation part 30 Movement distance calculation part 32 Deviation amount calculation Unit 34 camera posture determination unit 40 display device 42 display processing unit 44 distortion correction unit 46 linear movement determination unit 50 operation unit 100 vehicle-mounted camera posture detection device 200 calibration sheet

Claims (6)

An on-vehicle camera posture detection device that photographs a plurality of calibration indices included in a calibration sheet arranged around a vehicle with a camera mounted on the vehicle, and detects the posture of the camera based on the obtained image,
A pre-movement image storage means for storing an image of the calibration sheet photographed by the camera as a pre-movement image;
First posture estimation means for estimating the posture of the camera based on the pre-movement image stored in the pre-movement image storage means;
A post-movement image storage means for storing, as a post-movement image, an image of the calibration sheet taken by the camera after moving the vehicle in a straight line a predetermined distance;
Second posture estimation means for estimating the posture of the camera based on the post-movement image stored in the post-movement image storage means;
Sheet deviation amount calculating means for calculating a deviation amount of the calibration sheet from a predetermined position based on estimation results by the first and second posture estimating means;
Camera posture determination for determining the posture of the camera based on the posture estimated value estimated by at least one of the first and second camera posture estimating means and the deviation amount calculated by the sheet deviation amount calculating means. Means,
Linear movement determination means for determining whether or not the movement of the vehicle from the acquisition of the pre-movement image to the acquisition of the post-movement image has been made linear;
A vehicle-mounted camera posture detection device comprising: In claim 1,
By photographing through the lens with the camera, an image with a distorted periphery with respect to the center is obtained,
Further comprising distortion correction means for performing distortion correction on a plurality of images obtained by photographing with the camera at a plurality of locations corresponding to before and after movement of the vehicle and an intermediate position;
The in-vehicle camera attitude detection device, wherein the linear movement determination unit determines whether or not the vehicle has moved linearly based on a plurality of images after the distortion is removed by the distortion correction unit. . In claim 2,
The linear movement determination unit extracts feature points corresponding to the same part of the calibration sheet for the plurality of images after distortion removal, and each of the plurality of feature points included in the plurality of images is linear. An on-vehicle camera attitude detection device that determines that the movement of a vehicle is linear when it is arranged. In claim 3,
An in-vehicle camera attitude detection device further comprising display processing means for displaying an image on which distortion correction has been performed by the distortion correction means together with the plurality of feature points on a display device. In claim 4,
The on-vehicle camera attitude detection device characterized in that the image displayed on the display device by the display processing means includes a linear auxiliary line passing through both ends of the plurality of feature points. In any one of Claims 1-5,
The in-vehicle camera attitude detection device characterized in that the linear movement determination means notifies the driver of the vehicle when it is determined that the vehicle has not been moved linearly.

Images