JP2018098715A - On-vehicle camera posture change detection device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an on-vehicle camera posture change detection device and method capable of accurately detecting a posture change of a camera.SOLUTION: A mark 52 (virtual image 52') appearing on a windshield 14 inside a passenger compartment 12 is imaged so as to detect a current position (coordinate (u, v)) of a position of the mark 52 (virtual image 52'). A detected current position (u, v) is compared with a previously preserved initial position (coordinate (u0, v0)), so as to detect a posture change of a camera 18 by detecting a position change of the target 52 (virtual image 52').SELECTED DRAWING: Figure 6

Description

  The present invention relates to an on-vehicle camera posture change detection apparatus and method for detecting a posture change of a camera that photographs the front of a vehicle through a windshield.

  The vehicle is equipped with a camera that captures the front. For example, the camera is fixed in the vicinity of the rearview mirror, but the posture may change compared to the time of installation due to factors such as vehicle vibration, temperature change, and passenger contact. When the posture of the camera changes, an error exceeding the allowable amount occurs in the sensing information of the camera, which affects vehicle control using the sensing information.

  Patent Document 1 discloses an apparatus for detecting an optical axis shift (posture change) of a camera provided in a vehicle. In this apparatus, a mark index provided on a windshield or bonnet is photographed by a camera, and a displacement of the mark index in the image is detected to detect an optical axis shift of the camera.

JP 2004-173037 A

  When the mark index is provided on the windshield as in the apparatus of Patent Document 1, the camera and the mark index are too close to each other, and the camera is not focused on the mark index. For this reason, it is not practical to use a mark index provided on the windshield in order to detect an optical axis shift of the camera. On the other hand, when a mark index is provided on the bonnet, there are the following problems. The bonnet is a movable part, and the position of the mark index provided on the bonnet easily changes. For this reason, even if there is no deviation of the optical axis of the camera, the position of the mark index is not constant in the image taken by the camera. Therefore, the detection accuracy of the optical axis deviation is low.

  The present invention has been made in view of such problems, and an object thereof is to provide an on-vehicle camera posture change detection apparatus and method capable of detecting a camera posture change with high accuracy.

  A first aspect of the present invention is an in-vehicle camera attitude change detection device including a camera that captures the front of a vehicle through a windshield and an attitude change detection unit that detects an attitude change of the camera. The camera shoots the mark reflected on the windshield, and the posture change detection unit detects a change in the posture of the camera by detecting a change in the position of the mark in the image shot by the camera. It is characterized by detecting.

  In the above-described configuration, a mark in the passenger compartment reflected on the windshield is photographed, and a change in the posture of the camera is detected by detecting a change in the position of the mark in the photographed image. The landmark itself does not move. For this reason, when there is no change in the posture of the camera, the position of the mark coincides with the initial position in the image taken by the camera. On the other hand, when there is a change in the posture of the camera, the position of the mark is deviated from the initial position in the image captured by the camera. According to the above configuration, since the deviation from the initial position is detected, it is possible to increase the detection accuracy of the posture change of the camera.

  In the first invention, the mark may be provided at a position outside the angle of view of the camera. According to the above configuration, since the outside world is not obstructed by the landmark when taking a picture by the camera, more sensing information can be acquired.

  1st invention WHEREIN: It has the shielding member which can be opened and closed between the said windshield and the said mark, The said mark is reflected on the said windshield in the state by which the said shielding member was open | released, The state by which the said shielding member was obstruct | occluded In this case, it may be prevented from being reflected on the windshield. According to the above configuration, the mark can be reflected on the windshield when confirming the change in the posture of the camera, and the mark can be prevented from being reflected on the windshield at other times.

  In the first invention, the shielding member may be provided on a cover that covers the camera. According to the above configuration, since the shielding member is provided in the vicinity of the camera, even if the blocked member is reflected on the windshield, the image is not focused on the image. Therefore, the shielding member reflected on the windshield is not reflected in the image taken by the camera.

  In the first invention, a bonnet serving as a background may be provided when the mark reflected on the windshield is photographed by the camera. According to the above configuration, it is possible to easily recognize the mark when confirming the change in the posture of the camera.

  A second invention is a method for detecting a change in posture of an in-vehicle camera that detects a change in posture of a camera that captures the front of a vehicle through a windshield, and is used to mark a position in a vehicle interior reflected on the windshield within the angle of view of the camera. The position of the camera is detected by photographing the mark reflected on the windshield by the camera and detecting the position change of the mark in the image photographed by the camera.

  In the above-described configuration, a mark in the passenger compartment reflected on the windshield is photographed, and a change in the posture of the camera is detected by detecting a change in the position of the mark in the photographed image. Since the mark itself does not move, the position of the mark is fixed in the image captured by the camera when there is no change in the camera posture, while the mark is displayed in the image shot by the camera when there is a change in the camera posture. The position of is displaced. Therefore, according to the above configuration, it is possible to increase the detection accuracy of the posture change of the camera.

  In the second invention, the vehicle bonnet may be opened as a background, and the mark reflected on the windshield may be photographed by the camera. According to the above configuration, it is possible to easily recognize the mark when confirming the change in the posture of the camera.

  According to the present invention, it is possible to increase the detection accuracy of the posture change of the camera.

FIG. 1A is a configuration diagram of a vehicle including an on-vehicle camera posture change detection device according to the present embodiment, and FIG. 1B is a plan view of a mark. FIG. 2 is an explanatory diagram for explaining an original image and a recognized image. FIG. 3 is a flowchart of the initial posture detection process. FIG. 4 is a diagram showing the first recognition image. FIG. 5 is a flowchart of the posture change detection process. FIG. 6 is a diagram showing a second recognition image.

  DESCRIPTION OF EMBODIMENTS Hereinafter, an in-vehicle camera posture change detection apparatus and method according to the present invention will be described in detail with reference to the accompanying drawings by using preferred embodiments.

[1 Vehicle 10]
An in-vehicle camera posture change detection device 50 (hereinafter also referred to as “posture change detection device 50”) according to the present embodiment will be described with reference to FIG. 1A. The vehicle 10 is an automatic driving vehicle or a semi-automatic driving vehicle capable of automatically controlling at least one of the driving force device 30, the steering device 32, and the braking device 34.

  A seat (not shown) is provided in the passenger compartment 12 of the vehicle 10, a windshield 14 is provided opposite to the seat, and a dashboard 16 is provided below the windshield 14. One or a plurality of cameras 18 are provided in the passenger compartment 12 and above the windshield 14, for example, in the vicinity of a rearview mirror (not shown). The camera 18 is supported by a bracket (not shown) attached to the windshield 14. The camera 18 is covered with a cover 20. The cover 20 is fixed to the roof 22 in the passenger compartment 12. The posture of the camera 18 is adjusted so as to photograph the front F of the vehicle 10 through the windshield 14.

  The vehicle 10 includes an image recognition device 26, a vehicle control device 28, a driving force device 30, a steering device 32, a braking device 34, and a notification device 36.

  The image recognition device 26 includes one or more ECUs, and includes a first CPU 40, a first storage device 42, and the like. In the present embodiment, the recognition unit 44 is realized by the first CPU 40 executing the program stored in the first storage device 42. Note that the recognition unit 44 can also be realized by hardware including an integrated circuit or the like.

  As shown in FIG. 2, the recognition unit 44 defines a part of the original image Im_o photographed by the camera 18 as a recognition image Im_r. Before the vehicle 10 is shipped, the center positions of the original image Im_o and the recognized image Im_r are adjusted so as to coincide with the optical axis of the camera 18. The position of the recognized image Im_r in the original image Im_o is stored in the first storage device 42. The recognition unit 44 performs image recognition processing on the recognition image Im_r, recognizes the recognition objects 80 and 82 in the outside world, and recognizes the positions of the recognition objects 80 and 82. The recognition result of the recognition unit 44 is used by the vehicle control device 28 when the vehicle 10 is traveling, for example.

  The vehicle control device 28 includes one or more ECUs. In addition to the image recognition device 26, the vehicle control device 28 acquires various information from an external information acquisition device (not shown), a vehicle sensor, and the like, and controls acceleration / deceleration, steering, and braking of the vehicle 10. The external information acquisition device includes a radar, a navigation device, a communication device, and the like. The vehicle sensor includes a sensor that detects the behavior (vehicle speed, acceleration / deceleration, yaw rate, etc.) of the vehicle 10 and a sensor that detects an operation element (accelerator pedal, brake pedal, steering angle, etc.).

  The driving force device 30 includes a driving force ECU and a driving source of the vehicle 10 such as an engine and / or a driving motor. The driving force device 30 generates a traveling driving force (torque) for the vehicle 10 to travel in accordance with a control command output from the vehicle control device 28, and transmits the traveling driving force (torque) to the wheels via a transmission. The steering device 32 includes an EPS (electric power steering system) ECU and an EPS device. The steering device 32 changes the direction of the wheel (steering wheel) in accordance with a control command output from the vehicle control device 28. The braking device 34 is, for example, an electric servo brake that uses a hydraulic brake together, and includes a brake ECU and a brake actuator. The braking device 34 brakes the wheel in accordance with a control command output from the vehicle control device 28. The notification device 36 includes a notification ECU, a display device, and / or an acoustic device. The notification device 36 notifies various guides and warnings in accordance with a notification command output from the vehicle control device 28.

[2 Attitude change detection device 50]
In addition to the windshield 14, the camera 18, and the image recognition device 26 described above, the posture change detection device 50 according to the present embodiment includes a mark 52, a shielding member 54, and an inspection device 56.

  The marks 52 are formed by patterns, characters, shapes, colors, light sources, etc. that can be recognized when viewed from above the dashboard 16. As shown in FIG. 1B, the mark 52 of the present embodiment has a cross pattern as viewed from above. The mark 52 may be a part of the interior of the vehicle 10. The mark 52 is a position reflected on the windshield 14 within the angle of view 58 of the camera 18, that is, a position where the mark 52 reflected on the windshield 14 is photographed by the camera 18 and a position outside the angle of view 58 of the camera 18. Fixed to. When a plurality of cameras 18 are provided, the mark 52 is provided at a position reflected on the windshield 14 within the angle of view 58 of each camera 18. Specifically, the mark 52 is provided on the upper surface of the non-movable dashboard 16. Hereinafter, the mark 52 reflected on the windshield 14 is also referred to as a virtual image 52 ′. A sufficiently long optical path L is set between the mark 52 and the camera 18 so that the camera 18 is focused on the virtual image 52 ′.

  The shielding member 54 is provided so that the optical path L between the windshield 14 and the mark 52 can be opened and closed. In the present embodiment, the opening 20a of the cover 20 is disposed in the optical path L, and the shielding member 54 is provided so that the opening 20a can be opened and closed. The shielding member 54 may be provided on the dashboard 16. The shielding member 54 is opened or closed manually or automatically.

  The inspection device 56 is configured by a terminal device that can be operated by an operator, such as a personal computer or a tablet terminal, and includes a second CPU 60, a second storage device 62, a display device 68, an input device (not shown), and the like. In the present embodiment, the posture change detection unit 64 and the calibration unit 66 are realized by the second CPU 60 executing the program stored in the second storage device 62. Note that the posture change detection unit 64 and the calibration unit 66 can be realized by hardware including an integrated circuit. The inspection device 56 can communicate with the image recognition device 26 and the vehicle control device 28. The inspection device 56 is detachable from the in-vehicle network including the image recognition device 26 and the vehicle control device 28, but may be always connected.

  The posture change detection unit 64 has an initial position [coordinates (u0, v0)] (see FIGS. 4 and 6) and a current position [coordinates (u, v)] (see FIG. 4 and FIG. 6) reflected on the windshield 14. (See FIG. 6) to detect a change in posture of the camera 18 (see [3] and [4] below). The calibration unit 66 performs a calibration process based on the posture change information of the camera 18 detected by the posture change detection unit 64, and corrects the position of the recognition image Im_r defined by the recognition unit 44 (see [5] below).

[3 Initial posture detection processing]
The initial posture detection process will be described with reference to FIG. In the initial posture detection process shown in FIG. 3, a series of processes for storing the initial position and initial posture of the camera 18 are performed. The initial posture detection process is performed, for example, before shipment of the vehicle 10 when the camera 18 is installed in the passenger compartment 12. Note that the initial posture detection process may be performed after the calibration process of the camera 18.

  The operator opens the shielding member 54 before performing the initial posture detection process. The mark 52 does not appear on the windshield 14 within the angle of view 58 of the camera 18 when the shielding member 54 is closed, whereas it appears on the windshield 14 within the angle of view 58 of the camera 18 when the shielding member 54 is open. Furthermore, the operator opens the hood 70 of the vehicle 10 and uses it as the background of the first recognition image Im_r1 (see FIG. 4) that is captured by the camera 18. When the bonnet 70 is opened, the virtual image 52 ′ is easily recognized in the first recognition image Im_r1. However, the opening operation of the bonnet 70 is not essential. After the operator performs the above operations, the operator operates the inspection device 56 to start the initial posture detection process shown in FIG.

  In step S <b> 1, the camera 18 photographs the front of the vehicle 10. In step S <b> 2, the image recognition device 26 acquires the first original image Im_o <b> 1 captured by the camera 18. At this time, as illustrated in FIG. 4, the recognition unit 44 defines a part of the first original image Im_o1 as the first recognition image Im_r1.

  In step S <b> 3, the operator operates the inspection device 56 to designate an image of a region of interest (hereinafter also referred to as “ROI”) including the virtual image 52 ′ of the mark 52 from the first recognition image Im_r <b> 1 as the template 72 ( (See FIG. 4). The posture change detection unit 64 detects the position of the template 72, for example, the coordinates (u0, v0) of the upper left position of the template 72 as shown in FIG. The coordinates (u0, v0) correspond to the initial position of the mark 52 (virtual image 52 ') in the windshield 14.

  As another method of step S3, the posture change detection unit 64 may perform template matching to detect the coordinates (u0, v0) of the upper left position of the template 72. In this case, the same (similar) image information as that of the template 72 is stored in the second storage device 62 in advance.

  In step S <b> 4, the posture change detection unit 64 stores the ROI information, here, the image information and the coordinates (u0, v0) of the template 72 in the first storage device 42.

[4 Attitude change detection processing]
The posture change detection process will be described with reference to FIG. In the posture change detection process shown in FIG. 5, a series of processes for comparing the initial posture of the camera 18 with the current posture is performed. The first posture change detection process is performed after a certain amount of time has elapsed after the initial posture detection process shown in FIG. 3, and the second and subsequent posture change detection processes are performed to some extent after the previous posture change detection process. It will be done after the time. For example, the posture change detection process may be performed at the time of vehicle inspection or may be performed every predetermined time (period).

  As in the initial posture detection process, the operator opens the shielding member 54 before performing the posture change detection process. Further, the operator opens the hood 70 of the vehicle 10 and uses it as the background of the second recognition image Im_r2 (see FIG. 6) photographed by the camera 18. When the bonnet 70 is opened, the virtual image 52 ′ is easily recognized in the second recognition image Im_r2. However, the opening operation of the bonnet 70 is not essential. After the operator performs the above work, the operator operates the inspection device 56 to start the posture change detection process shown in FIG.

  In step S <b> 11, the camera 18 photographs the front of the vehicle 10. In step S12, the image recognition device 26 acquires the second original image Im_o2 photographed by the camera 18. At this time, as illustrated in FIG. 6, the recognition unit 44 defines a part of the second original image Im_o2 as the second recognition image Im_r2.

  In step S13, the posture change detection unit 64 detects the position of an area having image information similar to the template 72 (hereinafter referred to as “similar area 72 ′”) from the second original image Im_o2. Specifically, the image information of the template 72 is read from the second storage device 62 and template matching is performed. At this time, as shown in FIG. 6, the coordinates (u, v) of the upper left position of the similar region 72 ′ are detected. The coordinates (u, v) correspond to the current position of the mark 52 (virtual image 52 ′) in the windshield 14. By the way, when template matching is performed using an existing image processing technique, the measurement position is often obtained as an integer. For this reason, in this embodiment, template matching is performed to detect an integer part of coordinates (u, v), and then subpixel estimation is performed to detect a decimal part of coordinates (u, v).

  In step S <b> 14, the posture change detection unit 64 calculates the displacement amount of the template 72 (virtual image 52 ′). Here, the posture change detection unit 64 reads the coordinates (u0, v0) of the template 72 saved in step S4 of the initial posture detection process from the second storage device 62, and coordinates (u, v0) with respect to the coordinates (u0, v0). The displacement amount (u−u0, v−v0) and the displacement direction of v) are calculated. That is, the displacement amount and displacement direction of the current position (measurement position) with respect to the initial position (reference position) of the template 72 are calculated.

  In step S <b> 15, the posture change detection unit 64 determines the amount of displacement of the camera 18. Here, the posture change detection unit 64 reads the allowable threshold values (uth, vth) of the displacement amount of the template 72 from the second storage device 62. Then, the displacement amounts (u−u0, v−v0) calculated in step S14 are compared with the allowable threshold values (uth, vth). When | u−u0 |> uth or | v−v0 |> vth, that is, either the amount of displacement in the horizontal direction (vehicle width direction) or the amount of displacement in the vertical direction (vehicle vertical direction) is greater than the allowable threshold. If so (step S15: YES), the process proceeds to step S16. On the other hand, when | u−u0 | ≦ uth and | v−v0 | ≦ vth, that is, when both the horizontal displacement amount and the vertical displacement amount are small (step S15: NO), the process proceeds to step S17. To do.

  When the process proceeds from step S15 to step S16, the posture change detection unit 64 determines that the posture of the camera 18 has changed. At this time, the inspection device 56 performs a predetermined process corresponding to the posture change. For example, the display device 68 notifies that the posture change of the camera 18 has occurred. Alternatively, an attitude change detection signal is output to the vehicle control device 28. The vehicle control device 28 that has received the posture change detection signal restricts part or all of the automatic driving function. The posture change detection unit 64 stores the displacement (u−u0, v−v0) and the displacement direction in the second storage device 62 in preparation for the calibration process (see [5] below).

  When the process proceeds from step S15 to step S17, the posture change detection unit 64 determines that there is no posture change of the camera 18. Even if the amount of displacement (u−u0, v−v0) is not zero, the sensing information of the camera 18 is not greatly affected if the amount of displacement (u−u0, v−v0) is equal to or less than the allowable threshold (uth, vth). For this reason, displacement amounts (u−u0, v−v0) that are less than or equal to the allowable threshold values (uth, vth) are allowed.

[5 Calibration processing]
The calibration process will be described with reference to FIG. When it is determined that there is a posture change in the posture change detection process shown in FIG. 5 (step S16), the operator operates the inspection device 56 to perform a calibration process. Alternatively, the configuration processing may be automatically performed by the inspection device 56 executing a calibration processing program stored in the second storage device 62. The calibration unit 66 reads the displacement amount (u−u 0, v−v 0) and the displacement direction from the second storage device 62. Then, the position of the recognized image Im_r stored in the first storage device 42 of the image recognition device 26 is corrected by a displacement amount (u−u0, v−v0) in the direction opposite to the displacement direction.

[6 Summary of this embodiment]
The posture change detection device 50 according to the present embodiment is fixed inside the vehicle compartment 12, a camera 18 that captures the front of the vehicle 10 through the windshield 14, a posture change detection unit 64 that detects a posture change of the camera 18, and the like. A mark 52 is provided. The camera 18 captures a mark 52 (virtual image 52 ′) reflected on the windshield 14. The posture change detection unit 64 detects a change in the posture of the camera 18 by detecting a change in the position of the mark 52 (virtual image 52 ′) in the second recognition image Im_r 2 photographed by the camera 18.

  As described above, the posture change detection device 50 captures the mark 52 (virtual image 52 ′) in the passenger compartment 12 reflected on the windshield 14, and the position of the mark 52 (virtual image 52 ′) in the captured second recognition image Im_r2. By detecting the change, the posture change of the camera 18 is detected. The landmark 52 itself does not move. Therefore, when there is no change in the posture of the camera 18, the position (u, v) of the mark 52 (virtual image 52 ′) is the initial position [coordinates (u0, v0)] in the second recognition image Im_r2 photographed by the camera 18. Matches. On the other hand, when there is a change in the posture of the camera 18, the current position [coordinates (u, v)] of the mark 52 (virtual image 52 ′) in the second recognition image Im_r 2 captured by the camera 18 is the initial position [coordinates (u 0, v0)]. According to this configuration, since the deviation from the initial position [coordinates (u0, v0)] is detected, the detection accuracy of the posture change of the camera 18 can be increased.

  In the posture change detection device 50, the mark 52 is provided at a position outside the angle of view 58 of the camera 18. According to this configuration, the outside world is not obstructed by the mark 52 during shooting by the camera 18, so that more sensing information can be acquired.

  The posture change detection device 50 includes a shield member 54 that can be freely opened and closed between the windshield 14 and the mark 52. The mark 52 is reflected on the windshield 14 when the shielding member 54 is opened, and is not reflected on the windshield 14 when the shielding member 54 is closed. According to this configuration, the mark 52 can be reflected on the windshield 14 when confirming the posture change of the camera 18, and the mark 52 can be prevented from being reflected on the windshield 14 at other times.

  The shielding member 54 is provided on the cover 20 that covers the camera 18. According to this configuration, since the shielding member 54 is provided in the vicinity of the camera 18, even if the blocked member 54 is reflected on the windshield 14, the image of the camera 18 is not focused on the image. Therefore, the shielding member 54 reflected on the windshield 14 is not reflected in the first recognition image Im_r1 and the second recognition image Im_r2 captured by the camera 18.

  The posture change detection device 50 includes a bonnet 70 that serves as a background when the camera 18 captures a mark 52 (virtual image 52 ′) reflected on the windshield 14. According to this configuration, it is possible to easily recognize the mark 52 (virtual image 52 ′) when confirming the posture change of the camera 18.

  Further, in the posture change detection method according to the present embodiment, a mark 52 is provided at a position in the passenger compartment 12 reflected on the windshield 14 within the angle of view 58 of the camera 18, and the mark 52 (virtual image) reflected on the windshield 14 by the camera 18. 52 ′). Then, a change in the posture of the camera 18 is detected by detecting a change in the position of the mark 52 (virtual image 52 ′) in the second recognition image Im_r 2 photographed by the camera 18.

  In the posture change detection method, the hood 70 of the vehicle 10 is opened as a background, and a mark 52 (virtual image 52 ′) reflected on the windshield 14 is photographed by the camera 18.

  It should be noted that the on-vehicle camera posture change detection device 50 and method according to the present invention are not limited to the above-described embodiments, and can of course adopt various configurations without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Vehicle 12 ... Vehicle compartment 14 ... Windshield 18 ... Camera 20 ... Cover 26 ... Image recognition device 50 ... Posture change detection device 52 ... Mark 52 '... Virtual image 54 ... Shielding member 56 ... Inspection device 58 ... Angle of view 64 ... Posture Change detector 70 ... Bonnet

Claims (7)

A camera that captures the front of the vehicle through the windshield,
A posture change detection device for a vehicle-mounted camera comprising a posture change detection unit for detecting a posture change of the camera,
With a mark fixed in the passenger compartment,
The camera photographs the mark reflected on the windshield,
The posture change detection device for an in-vehicle camera, wherein the posture change detection unit detects a posture change of the camera by detecting a change in the position of the mark in an image photographed by the camera. The posture change detection device for an in-vehicle camera according to claim 1,
The in-vehicle camera posture change detection device, wherein the mark is provided at a position outside the angle of view of the camera. In the on-vehicle camera posture change detection device according to claim 1 or 2,
A freely openable and closable shielding member between the windshield and the mark;
The in-vehicle camera posture change detection device, wherein the mark is reflected on the windshield when the shielding member is opened, and is not reflected on the windshield when the shielding member is closed. The posture change detection device for an in-vehicle camera according to claim 3,
The in-vehicle camera posture change detection device, wherein the shielding member is provided on a cover that covers the camera. In the posture change detection apparatus of the vehicle-mounted camera of any one of Claims 1-4,
An in-vehicle camera posture change detection device, comprising: a bonnet as a background when the mark reflected on the windshield is photographed by the camera. An in-vehicle camera attitude change detection method for detecting an attitude change of a camera that images the front of a vehicle through a windshield,
A mark is provided at a position in the passenger compartment reflected on the windshield within the angle of view of the camera,
Shooting the landmark reflected on the windshield with the camera,
An in-vehicle camera posture change detection method, comprising: detecting a change in posture of the camera by detecting a change in position of the mark in an image photographed by the camera. The attitude change detection method for an in-vehicle camera according to claim 6,
An in-vehicle camera posture change detection method, wherein the vehicle hood is opened to make a background and the mark reflected on the windshield is photographed by the camera.

Images