JP2015106777A - Image processing system, operation support device, navigation device, and camera device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system capable of improving visibility of an image displayed on the screen of a display device.SOLUTION: An image processing system 24 comprises: a generator 25 which, on the basis of an imaging signal for identifying an image captured by an in-vehicle camera 16, generates an image display signal for identifying an image displayed on the screen of a display device 15; and a processor 26 which performs coordinate transformation on an image of the imaging signal in accordance with a posture information signal indicating variation in the posture of a vehicle relative to the ground surface. Even when the posture of the vehicle varies relative to the ground surface, the inclination of a subject is corrected.

Description

  The present invention relates to an image processing apparatus.

  Patent Document 1 discloses an on-vehicle camera calibration device. The in-vehicle camera is fixed to the vehicle. The calibration plate is imaged by the in-vehicle camera. A calibration pattern is drawn on the calibration plate. The position and orientation of the in-vehicle camera are specified based on the calibration pattern image. The calibration parameters of the in-vehicle camera are specified according to the position and orientation. The deviation of the subject in the image is corrected according to the calibration parameter. Thus, the calibration of the in-vehicle camera is performed.

JP 2010-103730 A JP 2001-328484 A

  Once determined, the calibration parameters are fixed. There is no change to the in-vehicle camera. Here, when the vehicle is tilted with respect to the ground on which the vehicle is disposed, the in-vehicle camera is similarly tilted. The subject tilts in the image captured by the in-vehicle camera. On the other hand, the vehicle can only move parallel to the ground where the car is located. As a result, a direction shift occurs between the actual movement of the vehicle and the movement of the subject in the screen. The direction of movement does not match. The driver cannot satisfactorily combine the movement of the vehicle and the movement of the subject in the screen. The visibility of the subject is reduced.

  According to some aspects of the present invention, an image processing device capable of improving the visibility of an image displayed on a screen of a display device is provided.

  In one embodiment of the present invention, a generation unit that generates an image display signal that specifies an image displayed on a screen of a display device based on an imaging signal that specifies an image outside the vehicle that is captured by a vehicle-mounted camera, and a vehicle are arranged. The present invention relates to an image processing apparatus including a processing unit that performs coordinate transformation on an image of the imaging signal in accordance with a posture information signal indicating a posture change of a vehicle with respect to the ground.

  The in-vehicle camera is fixed to the vehicle. When the attitude of the vehicle changes with respect to the ground on which the vehicle is placed, the attitude of the in-vehicle camera changes similarly with respect to the ground. Accordingly, the subject is tilted in the image captured by the in-vehicle camera. At this time, if coordinate transformation is performed on the image, the inclination of the subject can be corrected. As a result, the image is displayed on the screen of the display device based on the straight subject without tilting with respect to the ground on which the vehicle is placed. The visibility of the image is definitely increased. The driver can satisfactorily link the movement of the vehicle and the movement of the subject in the screen.

  The coordinate conversion can convert an angle around an axis of an orthogonal triaxial coordinate system set in the vehicle. In this way, the angle conversion is reliably realized.

  The image processing apparatus further includes a storage unit, and the generation unit can perform image processing on the imaging signal using a processing parameter stored in the storage unit when generating the image display signal. Can change the processing parameter in accordance with the posture information signal. In this way, the angle conversion is reliably realized.

  The image processing may be overhead conversion. Since the inclination of the subject is corrected in the image of the imaging signal, the subject can be projected with a line of sight perpendicular to the ground on which the vehicle is placed in the image after the overhead conversion.

  The image processing apparatus as described above can be incorporated in the driving support apparatus. At this time, the driving support device includes an image processing device. Such a driving support device can be used by being mounted on a vehicle, for example.

  In addition, the image processing apparatus as described above can be incorporated in a navigation apparatus. At this time, the navigation device includes an image processing device. Such a navigation device can be used by being mounted on a vehicle, for example.

  In addition, the image processing apparatus as described above can be incorporated in the camera apparatus. At this time, the camera device includes a data conversion device. Such a camera device can be used by being mounted on a vehicle.

  As described above, according to the disclosed device, the visibility of the image displayed on the screen of the display device is enhanced.

1 is a perspective view schematically showing an overall image of a vehicle, that is, an automobile according to an embodiment. It is a block diagram which shows the structure of a driving assistance device roughly. It is a figure which shows an example of the image displayed on the screen of a display panel. It is a figure which shows an example of the image imaged with a camera apparatus. It is a figure which shows the other specific example of the image imaged with a camera apparatus. It is a figure which shows the other specific example of the image imaged with a camera apparatus. It is a figure which shows the other specific example of the image displayed on the screen of a display panel. It is a figure which shows the other specific example of the image imaged with a camera apparatus. It is a figure which shows the other specific example of the image displayed on the screen of a display panel. It is a figure which shows the other specific example of the image imaged with a camera apparatus. It is a perspective view which shows roughly the whole image of the vehicle which concerns on other embodiment, ie, a motor vehicle.

  Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 schematically shows an automobile 11 according to an embodiment. A driving support device 13 is mounted on the vehicle 12 of the automobile 11. The driving support device 13 includes a navigation device 14. The navigation device 14 has a display panel (display device) 15. The display panel 15 can be incorporated in the dashboard of the vehicle 12, for example. A map and other information can be displayed on the display screen of the display panel 15.

  The driving support device 13 includes a camera device 16. The camera device 16 is installed in the rear part of the vehicle 12, for example. The camera device 16 is attached above the rear bumper of the vehicle 12, for example. The optical axis 17 of the camera device 16 is set at a predetermined angle with respect to the horizontal plane, for example. The optical axis 17 is inclined so as to approach the ground on which the vehicle 12 is placed as it moves backward from the vehicle 12. The camera device 16 images a subject behind the vehicle 12. Here, a captured image of the subject is acquired in an angle of view range of at least ± 90 ° in both the horizontal and vertical directions with respect to the optical axis 17.

  The camera device 16 generates an image signal of the display image (hereinafter referred to as “display image signal”) based on the captured image. The display image signal specifies an image of the camera viewpoint with respect to the rear of the vehicle 12 and an overhead image of the ground on which the vehicle 12 is arranged. The image or the overhead image may be switched arbitrarily. The camera device 16 is connected to the navigation device 14. A display image signal is supplied from the camera device 16 to the navigation device 14. An image behind the vehicle 12 can be displayed on the display screen of the display panel 15 based on the display image signal. In this driving support device 13, for example, the driver can check the state behind the vehicle 12 when entering the parking space backward in a parking lot. In this way, the driving support device 13 can support the driver's backward confirmation. The vehicle 12 includes a door 18. The door 18 is opened and closed for passengers to get on and off.

  As shown in FIG. 2, the camera device 16 includes a fisheye lens 21. The fish-eye lens 21 defines, for example, an incident surface of a subject with a hemispherical surface having an equator plane orthogonal to the optical axis 17. As a result, the light reflected from the subject is incident on the optical axis 17 in the horizontal and vertical directions in an angle of view range of at least ± 90 °. In the image, the subject shrinks away from the center of the circle.

  The camera device 16 includes an image sensor 22. The image sensor 22 converts an image of a subject projected from the fisheye lens 21 into an electric signal. The image sensor 22 includes pixels arranged in a matrix. The position of each pixel is specified according to a two-dimensional coordinate system set on the imaging surface of the image sensor 22. For example, in the Bayer array, the color of the subject is expressed by an electrical signal of RGGB (red green green blue).

  The camera device 16 includes an image processing unit 23. The image processing unit 23 is electrically connected to the image sensor 22. The image processing unit 23 reads out the electrical signals of the pixels while sequentially specifying the pixel positions, that is, addresses. The image processing unit 23 outputs a digital imaging signal, that is, imaging data. The imaging data specifies image information for each pixel.

  An image processing device 24 is incorporated in the camera device 16. The image processing device 24 receives imaging data from the image processing unit 23. The image processing device 24 includes a display image generation unit 25 and a processing unit 26. The display image generation unit 25 generates an image display signal that specifies an image displayed on the screen of the display panel 15 based on the imaging data. The display image generation unit 25 performs overhead conversion on the imaging data using the processing parameters when generating the image display signal. The processing parameter may be acquired from the storage unit 27, for example. At this time, a plurality of fixed processing parameters may be stored in the storage unit 27 in advance, or calculation is performed by the processing unit 26 according to an attitude information signal indicating the attitude change of the vehicle 12 with respect to the ground on which the vehicle 12 is placed. The processed parameters may be stored. In the storage unit 27, coordinate values on the screen are designated for each pixel of the imaging data. An overhead image is generated by the overhead conversion.

  The processing unit 26 performs coordinate conversion on the image of the imaging signal in accordance with the attitude information signal. The processing unit 26 changes the processing parameter according to the attitude information signal. According to the changed portion, the inclination of the subject with respect to the ground on which the vehicle 12 is arranged in the screen is eliminated. The processing parameter is changed according to the attitude of the vehicle 12. The storage unit 27 may store the processing parameters on a one-to-one basis for each tilt angle, or the processing unit 26 may calculate the processing parameters according to the posture.

  The processing unit 26 is connected to a detection sensor 28 provided for each wheel (right front wheel FR, left front wheel FL, right rear wheel RR, left rear wheel RL). Signal lines of the individual detection sensors 28 are connected to the connector 29. The detection sensor 28 detects the sinking amount of the suspension 31 attached to the corresponding wheels FR, FL, RR, RL. The attitude of the vehicle 12 is output as an attitude information signal according to the amount of sinking for each of the wheels FR, FL, RR, and RL. Here, for example, the origin of the orthogonal triaxial coordinate system is applied to the center of gravity of the vehicle 12 alone. One axis (for example, z-axis) of the orthogonal triaxial coordinate system defines the front-rear direction of the vehicle 12 parallel to the ground on which the vehicle 12 is disposed, and one axis (for example, x-axis) is parallel to the ground on which the vehicle 12 is disposed. The vehicle width direction of the vehicle 12 is defined, and one axis (for example, the y-axis) defines a direction orthogonal to the ground on which the vehicle 12 is disposed. The posture of the vehicle 12 is specified by, for example, the rotation angle around each axis of the orthogonal triaxial coordinate system. The processing parameter is set to a value that cancels these rotation angles.

  A lock sensor 32 is connected to the processing unit 26. The signal line of the lock sensor 32 is connected to the connector 33. The lock sensor 32 detects the operation of the door lock 34. Locking and unlocking of the door lock 34 is realized electrically, for example, through radio or optical communication. When the door lock 34 detects the unlocking operation, it outputs a detection signal. The processing unit 26 acquires an initial value of the detection sensor 28 in response to reception of the detection signal. When the door lock 34 is released after the engine of the automobile 11 is stopped and the key is removed and a predetermined time (for example, 10 minutes) has elapsed, it is estimated that nobody is in the automobile 11. The 12 postures correspond to the reference postures. According to the output of the detection sensor 28, the posture change of the vehicle 12 from the reference posture can be specified.

  The navigation device 14 includes an arithmetic processing unit 36. The arithmetic processing unit 36 is electrically connected to the image processing device 24. The arithmetic processing unit 36 can have, for example, an MPU (microprocessor unit) and a memory. For example, the MPU can operate according to software stored in the memory. The arithmetic processing unit 36 can acquire image information after the coordinate conversion is performed by the image processing device 24 from the camera device 16 in accordance with processing executed by software. The arithmetic processing unit 36 can switch between the navigation operation and the driving support operation of the navigation device 14. For example, when the gear of the automobile 11 is put in the back, the arithmetic processing unit 36 can switch the navigation device 14 from the navigation operation to the driving support operation.

  The display panel 15 is electrically connected to the arithmetic processing unit 36. Image information is supplied from the arithmetic processing unit 36 to the display panel 15. Depending on the supplied image information, a map or a video behind the vehicle 12 can be displayed on the display screen of the display panel 15.

  Next, the operation of the driving support device 13 will be briefly described. When an occupant or the like gets into the automobile 11, the door 18 is closed, and the engine is started, the detection sensor 28 specifies the initial value of the sinking amount. The processing unit 26 detects the inclination of the vehicle 12 based on the amount of deviation from the initial value. The processing unit 26 calculates a rotation angle around each axis of the three-dimensional coordinate system set for the vehicle 12. The processing unit 26 specifies the processing parameter in a form that cancels the calculated rotation angle. The processing unit 26 changes the processing parameter according to the change in the rotation angle.

  Now, for example, a case is assumed where an occupant gets on the rear seat and the rear of the vehicle 12 sinks. At this time, the processing unit 26 detects the tilt (tilt angle) of the vehicle 12 around the x axis and outputs it as an attitude information signal. The processing parameter is changed according to the output attitude information signal. Here, when the gear of the automobile 11 is put in the back gear, the arithmetic processing unit 36 switches the navigation device 14 from the navigation operation to the driving support operation. Based on the processing parameters and the attitude information signal stored in the storage unit 27, the inclination of the camera line of sight is corrected in the overhead image. The correction may be performed by storing a large number of processing parameters in the storage unit 27 and selecting them by the processing unit 26 based on the attitude control signal. Alternatively, the processing parameters stored in the storage unit 27 as initial values in advance The value calculated by the processing unit 26 based on the information signal may be added or subtracted. As shown in FIG. 3, a bird's-eye view image is displayed on the screen of the display panel 15 based on a straight camera line of sight without tilting with respect to the ground on which the vehicle 12 is placed. The visibility of the image is increased. Therefore, the driver can satisfactorily link the movement of the vehicle 12 and the movement of the subject in the screen. The driver can easily check the parallelism between the vehicle 12 and the parking line. The driver can satisfactorily stop the automobile 11 at the specified position in the parking space from the back. On the other hand, in the image captured by the camera device 16, the optical axis 17 of the camera device 16 is inclined downward, so that the straight line parallel to the y-axis is simultaneously reduced as shown in FIG. As the vehicle 12 moves away from the vehicle 12, the vehicle tilts away from each other. If such an image is displayed on the screen of the display panel 15 as it is, it is difficult to confirm the parallelism between the vehicle 12 and the parking line. It is difficult to go back to the specified position in the parking space. This hinders support for parking operations.

  For example, when a large number of people enter the passenger seat side of the automobile 11 in the front and rear seats, the processing unit 26 outputs the inclination (roll angle) of the vehicle 12 around the z axis as an attitude information signal. The processing parameter is changed according to the output attitude information signal. Based on the processing parameters in the storage unit 27, the inclination of the camera line of sight is corrected in the overhead view image. As described above, the overhead image is projected on the screen of the display panel 15 based on a straight camera line of sight without tilting with respect to the ground on which the vehicle 12 is placed. On the other hand, in the image captured by the camera device 16, the perspective of the subject is depicted on the left and right of the screen as shown in FIG. If such an image is displayed on the screen of the display panel 15 as it is, it is difficult for the driver to associate the steering operation with the movement of the subject in the screen. It is difficult to go back to the specified position in the parking space. This hinders support for parking operations.

  In addition, the processing unit 26 outputs the inclination (pan angle) of the vehicle 12 about the y axis as an attitude information signal according to the occupant's boarding position and the setting of the suspension 31. The processing parameter is changed according to the output attitude information signal. Based on the processing parameters in the storage unit 27, the inclination of the camera line of sight is corrected in the overhead view image. As described above, the overhead image is projected on the screen of the display panel 15 based on a straight camera line of sight without tilting with respect to the ground on which the vehicle 12 is placed. On the other hand, in the image captured by the camera device 16, as shown in FIG. 6, the straight traveling direction of the vehicle 12 is depicted tilted from the left and right center lines. If such an image is displayed on the screen of the display panel 15 as it is, it is difficult for the driver to associate the steering operation with the movement of the subject in the screen. It is difficult to go back to the specified position in the parking space. This hinders support for parking operations.

  The display image generation unit 25 may perform guide line superimposing processing instead of the above-described overhead conversion. As shown in FIG. 7, a guide line (dotted line in the figure) 38 is drawn in parallel with the horizon in the screen. The guide line 38 indicates a specific position on the ground where the vehicle 12 is disposed. In FIG. 7, the white tape 39 is pasted at a position corresponding to the guide line 38 in the actual parking space. The white tape 39 is imaged as part of the subject. The processing unit 26 detects a change in the attitude of the vehicle 12 with respect to the ground on which the vehicle 12 is arranged, as described above. The processing parameter is changed according to the detected inclination. Based on the processing parameters of the storage unit 27, the inclination of the ground on which the vehicle 12 is arranged is corrected in the image. A perspective image is projected on the screen of the display panel 15 with a straight line of sight of the camera without tilting with respect to the ground. On the other hand, in the image captured by the camera device 16, as shown in FIG. 8, the ground on which the vehicle 12 is disposed is depicted with an inclination with respect to the horizontal line of the screen. The guide line 38 cannot overlap the horizontal direction of the ground on which the vehicle 12 is disposed. If such an image is displayed on the screen of the display panel 15 as it is, it is difficult for the driver to associate the steering operation with the movement of the subject in the screen. It is difficult to go back to the specified position in the parking space. This hinders support for parking operations.

  In the display image generation unit 25, instead of the above-described overhead view conversion, the image may be converted into an image of a horizontal camera line of sight. In the picked-up image picked up by the fisheye lens 21, the subject expands as it moves away from the optical axis 17 toward the edge of the screen. As a result, the linear shape of the subject orthogonal to the horizontal plane in real space and the linear shape of the subject orthogonal to the vertical plane including the optical axis 17 are impaired, and the linear shape is curved. In particular, when the optical axis 17 is tilted downward, the ground on which the vehicle 12 is placed is curved, the building beside the road is tilted, and the subject is depicted unnaturally. As shown in FIG. 9, when the straight line shape of the subject is maintained in the vertical direction of the screen while maintaining straightness from the lower edge of the screen toward the center of the horizon, the image of the horizontal camera line of sight is can get. In such conversion, the processing unit 26 detects a change in the attitude of the vehicle 12 with respect to the ground on which the vehicle 12 is arranged, as described above. The processing parameter is changed according to the detected inclination. Based on the processing parameters in the storage unit 27, the inclination of the ground is corrected in the image. The perspective image is displayed on the screen of the display panel 15 with a straight camera line of sight without tilting with respect to the ground on which the vehicle 12 is placed. On the other hand, in the image captured by the camera device 16, as shown in FIG. 10, the ground on which the vehicle 12 is disposed is depicted with an inclination with respect to the horizontal line of the screen. Similarly, a vertical line perpendicular to the horizontal line is depicted tilted. If such an image is displayed on the screen of the display panel 15 as it is, it is difficult for the driver to associate the steering operation with the movement of the subject in the screen. It is difficult to go back to the specified position in the parking space. This hinders support for parking operations.

  As shown in FIG. 11, the camera device 16 may image a subject in front of the vehicle 12. Such captured images can be used, for example, for automatic driving of the automobile 11 or a brake support system. For example, the environment in front of the vehicle 12 can be grasped based on the image recognition technology. The camera device 16 may be mounted before and after the vehicle 12.

  The lens of the camera device 16 does not necessarily have to be the fisheye lens 21 as described above, and a central projection type lens may be used. A weight sensor incorporated in the seat can be used as the detection sensor 28 that detects the inclination of the vehicle 12. For example, if a seat sensor used for a warning signal for wearing a seat belt is used to detect the inclination of the vehicle 12, the cost for realizing the driving support device 13 can be reduced. In addition, the initial value of the detection sensor 28 does not need to be specified according to the unlocking of the door lock 34 when the engine is stopped, and may be stored in advance in the processing unit 26 as an absolute value for each individual detection sensor 28. . Further, the image processing device 24 can be incorporated not only in the camera device 16 as described above, but also may be configured separately from the camera device 16 or may be incorporated in the navigation device 14.

  DESCRIPTION OF SYMBOLS 12 Vehicle, 13 Driving assistance apparatus, 14 Navigation apparatus, 15 Display apparatus (display panel), 16 Car-mounted camera (camera apparatus), 24 Image processing apparatus, 25 Display image production | generation part, 26 Processing part.

Claims (7)

A generating unit that generates an image display signal that specifies an image displayed on the screen of the display device, based on an imaging signal that specifies an image outside the vehicle captured by the in-vehicle camera;
An image processing apparatus comprising: a processing unit that performs coordinate conversion on an image of the imaging signal in accordance with a posture information signal indicating a posture change of the vehicle with respect to a ground on which the vehicle is disposed.   The image processing apparatus according to claim 1, wherein the coordinate conversion converts an angle around an axis of an orthogonal triaxial coordinate system set in the vehicle.   The image processing apparatus according to claim 2, further comprising a storage unit that stores a processing parameter, wherein the generation unit uses the processing parameter stored in the storage unit when generating the image display signal. An image processing apparatus, wherein the processing unit has a function of changing the processing parameter in accordance with the posture information signal input to the processing unit.   The image processing apparatus according to claim 3, wherein the image processing is overhead conversion.   A driving support apparatus comprising the image processing apparatus according to claim 1.   A navigation apparatus comprising the image processing apparatus according to claim 1.   A camera apparatus comprising the image processing apparatus according to claim 1.