JP2016189576A - Image display control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display control device that can smoothen tied portions of plural images for display when the plural images are combined to provide a wide-angle image.SOLUTION: An image display control device includes: an acquisition unit for acquiring a first backward image obtained by imaging the back side from a rear portion of a vehicle body, a second backward image obtained by imaging the right back side containing a right rear portion of the vehicle body, and a third backward image obtained by imaging a left back side containing a left rear portion of the vehicle body; a detection unit for detecting the presence or absence of an obstacle in the first backward image, the second backward image, and the third backward image; a conversion unit for converting the first backward image, the second backward image and the third backward image by using a homography calculated based on the distance from the rear portion of the vehicle body to the obstacle when the obstacle is detected; an image composite unit for combining the converted first backward image, second backward image and third backward image to generate a composite image in a panoramic mode with the rear end portion of the vehicle body as a viewpoint; and a controlling unit for controlling to display the composite image on a display device.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image display control device.

  2. Description of the Related Art Conventionally, there has been proposed a vehicle periphery monitoring device that displays an image captured by an in-vehicle camera on a display device and presents a state around the vehicle to a driver. In such a conventional vehicle periphery monitoring device, it is desirable to provide as much information as possible when presenting the situation around the vehicle (the host vehicle or the vehicle body) to the driver. For example, when providing an image of the rear of the vehicle, if a wide-angle image (panoramic image) that includes not only the rear of the vehicle but also the rear of the left side and the rear of the right side can be provided, a wide range of situations can be recognized in a short time with a small amount of eye movement be able to.

Japanese Patent No. 5143209 JP 2009-81664 A

  When displaying such a panoramic image, an image directly behind the vehicle body captured by the rear camera, an image captured behind the left side captured by the left side camera, and an image captured behind the right side captured by the right side camera. Although the images are combined, it is preferable to display a panoramic image in which the entire obstacle is displayed and the joints between the captured images are smoothed when the obstacle is present.

  Accordingly, one of the problems of the present invention is that when a plurality of images are combined to provide a wide-angle image (panoramic image), the entire obstacle is displayed and the joints of the plurality of images are displayed smoothly. It is to provide an image display control device capable of performing the above.

  The image display control device of the present invention includes a first rear image obtained by imaging the rear from the rear part of the vehicle body, a second rear image obtained by imaging the right rear including the right side of the vehicle body, and the left side of the vehicle body. A third rear image obtained by imaging the left rear, a detection unit for detecting presence or absence of an obstacle in the first rear image, the second rear image, and the third rear image; When the obstacle is detected, the first rear image, the second rear image, and the third rear image are calculated using the homography calculated based on the distance from the rear portion of the vehicle body to the obstacle. And a rear end of the vehicle body by combining the first rear image, the second rear image, and the third rear image converted by the conversion unit. An image composition unit that generates a panoramic composite image as a viewpoint; and And a control unit that performs control to display the formed image on the display device. With this configuration, as an example, the joints in the composite image can be smoothed and the entire obstacle can be displayed on the composite image.

  Further, in the image display control device of the present invention, the conversion unit may be configured such that the shorter the distance, the wider the angle of view of the first rear image, and the second rear image. And the third rear image are converted. With this configuration, as an example, the joints in the composite image can be smoothed and the entire obstacle can be displayed on the composite image.

  In the image display control device of the present invention, a storage unit that stores a homography in which the angle of view of the first rear image becomes wider as the distance is shorter, and a homography table in which the distance is associated A distance measuring unit that measures a distance from the rear portion of the vehicle body to the obstacle, and the conversion unit performs a homography corresponding to the distance measured by the distance measuring unit from the homography table. Using the acquired homography, the first back image, the second back image, and the third back image are converted. With this configuration, as an example, the joints in the composite image can be smoothed so that the entire obstacle can be displayed on the composite image, and the load of the composite image generation processing while the vehicle is traveling can be reduced.

  Further, in the image display control device of the present invention, when the obstacle is detected, an overlapping area between the first rear image and the second rear image, the first rear image, and the third rear image. A homography calculating unit that assigns a calibration index to an overlapping area with a rear image and calculates the homography based on the calibration index and the position of the imaging unit, and the conversion unit includes: The first back image, the second back image, and the third back image are converted using the homography calculated by the homography calculation unit. With this configuration, as an example, even when the distance to the obstacle changes while the vehicle is traveling, the joint in the composite image can be smoothed and the entire obstacle can be displayed more reliably on the composite image.

  In the image display control device according to the aspect of the invention, when the plurality of obstacles are detected, the conversion unit acquires a homography corresponding to the distance for each of the detected obstacles. The first rear image, the second rear image, and the third rear image are converted using a graphic, and the image composition unit generates the composite image for each of the plurality of obstacles, The control unit performs control for switching the composite image for each of the plurality of obstacles and displaying the composite image on the display device. With this configuration, as an example, even when a plurality of obstacles are detected during traveling, a plurality of composite images obtained by combining the first rear images of the angle of view according to the distance to each obstacle The whole can be displayed.

FIG. 1 is a plan view illustrating an example of an arrangement location and an imaging range of an imaging unit in a vehicle body on which the image display control device according to the first embodiment is mounted. FIG. 2 is a side view showing an example of an arrangement location and an imaging range of an imaging unit in a vehicle body on which the image display control apparatus according to the first embodiment is mounted. FIG. 3 is a block diagram illustrating an example of an image display system including the image display control apparatus according to the first embodiment. FIG. 4 is a block diagram illustrating an example of a configuration of an image display control unit realized in the ECU of the image display control device according to the first embodiment. FIG. 5 is a diagram illustrating an example of a homography table according to the first embodiment. FIG. 6 is a diagram illustrating an example of a calibration index pattern according to the first embodiment. FIG. 7 is a diagram for explaining an example of the homography calculation method according to the first embodiment. FIG. 8 is a plan view illustrating an example of an imaging range in a vehicle body on which the image display control device according to the first embodiment is mounted. FIG. 9 is a diagram illustrating an example of a cut-out image cut out from a rear image captured by the image capturing unit of the image display control apparatus according to the first embodiment. FIG. 10 is a diagram illustrating an example of a panoramic image synthesized from a rear image captured by the imaging unit of the image display control apparatus according to the embodiment. FIG. 11 is a diagram illustrating an example of a cut-out image cut out from a conventional rear image. FIG. 12 is a diagram illustrating an example of a panoramic image synthesized from a conventional rear image. FIG. 13 is a diagram for explaining an example of homography switching according to the first embodiment. FIG. 14 is a diagram illustrating an example of a cutout image cut out from the rear image according to the first embodiment. FIG. 15 is a diagram illustrating an example of a panoramic image synthesized from the rear image according to the first embodiment. FIG. 16 is a flowchart illustrating an example of a procedure of panorama image generation processing according to the first embodiment. FIG. 17 is a flowchart illustrating an example of a procedure of panoramic image generation processing according to the second embodiment. FIG. 18 is a flowchart illustrating an example of a procedure of panorama image generation processing according to the third embodiment.

  Hereinafter, embodiments and modifications will be disclosed. The configurations of the embodiments and modified examples described below, and the operations, results, and effects brought about by the configurations are examples. The present invention can be realized by configurations other than those disclosed in the following embodiments and modifications. Further, according to the present invention, it is possible to obtain at least one of various effects and derivative effects obtained by the configuration.

  In addition, the same components are included in the embodiments and examples disclosed below. In the following, common reference numerals are given to similar components, and redundant description is omitted.

(Embodiment 1)
FIG. 1 is a plan view illustrating an example of an arrangement position and an imaging range of the imaging unit 2 in a vehicle body 1 (vehicle) on which the image display control apparatus according to the first embodiment is mounted. FIG. 2 is a side view of the vehicle body 1. The vehicle body 1 may be, for example, an automobile (an internal combustion engine automobile) that uses an internal combustion engine (engine, not shown) as a drive source, or an automobile (electric automobile) that uses an electric motor (motor, not shown) as a drive source. Or a fuel cell vehicle. Moreover, the motor vehicle (hybrid vehicle) which uses those both as a drive source may be sufficient. The vehicle body 1 is equipped with a plurality of imaging units 2 constituting an image display system 100 (see FIG. 3) together with the image display control device of the present embodiment. The vehicle body 1 of the present embodiment includes, for example, three imaging units 2a, 2b, and 2c. The imaging unit 2 is a digital camera including an imaging element such as a CCD (charge coupled device) or a CIS (CMOS image sensor). The imaging unit 2 can output image data, that is, moving image data at a predetermined frame rate.

  The imaging unit 2 includes, for example, a wide-angle lens or a fisheye lens so that a wide range of imaging can be performed. The imaging unit 2a mainly captures the rear from the rear part 1a of the vehicle body 1 and obtains a first rear image. As shown in FIGS. 1 and 2, the imaging unit 2 a is disposed at a position almost at the center of the rear portion 1 a of the vehicle body 1 at a predetermined height (for example, 1 m) from the road surface and hardly affected by rainfall or snowfall. . The imaging unit 2a has, for example, θ1 (approximately 180 °) as shown by a broken line in FIG. 1 and has a vertical angle of view of θ4 (for example, slightly upward as shown in FIG. 2). Approximately 90 °). Accordingly, the first rear image captured by the imaging unit 2a is distorted (curved) at the peripheral portion, but the following vehicle or parked vehicle, pedestrian or other obstacle, road surface state, road side state, etc. The information of almost the entire area behind can be acquired.

  The imaging unit 2b mainly captures the right rear including the right side of the vehicle body 1 to obtain a second rear image. The imaging unit 2b is fixed or built in the right door mirror 4a, for example. Similarly, the imaging unit 2c mainly captures the left rear including the left side of the vehicle body 1 to obtain a third rear image. The imaging unit 2c is fixed or built in the left door mirror 4b, for example. As shown in FIG. 1, the angle of view in the horizontal direction of the imaging unit 2b is, for example, θ2 (approximately 80 °). Similarly, as shown in FIG. 1, the horizontal field angle of the imaging unit 2b is, for example, θ3 (approximately 80 °). Further, as shown in FIG. 2, the angle of view in the vertical direction of the imaging unit 2c (2b) is, for example, θ5 (approximately 80 °) facing slightly upward. Note that the imaging range of the imaging unit 2b is set so as to include the right side body 1b of the vehicle body 1 in the second rear image. Similarly, the imaging range of the imaging unit 2c is set so that the left side body 1c of the vehicle body 1 is included in the third rear image. Thus, by capturing a rear image so as to include the side body, distance information (distance information) between the vehicle body 1 and an object existing on the side of the vehicle body 1, for example, another vehicle, a pedestrian, an obstacle, or the like. ) Can be included in the image. The imaging range illustrated in FIGS. 1 and 2 is an example, and the imaging range by the imaging unit 2 is not limited to the examples in FIGS. 1 and 2. The positions where the imaging units 2a, 2b, and 2c are mounted are only examples, and a first rear image obtained by imaging the rear from the rear part of the vehicle body 1, and a second rear image obtained by imaging the right rear including the right side of the vehicle body 1. If a third rear image obtained by imaging the left rear including the left side of the vehicle body 1 can be acquired, the mounting position can be changed as appropriate. Moreover, the freedom degree of adjustment of a mounting position and adjustment of an imaging range can be improved by changing the angle of view of the imaging units 2a, 2b, and 2c.

  As shown in FIG. 3, the image display system 100 mounted on the vehicle body 1 includes an ECU 11 (electronic control unit) that controls an image displayed on the display unit 10 a as the display device 10. The ECU 11 is an example of a display control unit or an image display control device. The display unit 10a is provided in place of, for example, a rear-viewing rear-view mirror provided at the front and upper part of the vehicle interior. On the display unit 10a, a panoramic composite image (panoramic image) with the rear end of the vehicle body 1 as a viewpoint, as exemplified in FIG. 7 described later, based on the image captured by the imaging unit 2, It is displayed in a state resembling a mirror image reflected in the room mirror. A user such as a driver can use the display unit 10a as a room mirror or instead of a room mirror.

  In the case of the vehicle body 1 provided with a conventional room mirror, the display device 10 (display unit 10a) having a shape substantially equivalent to the shape of the room mirror covers, for example, the mirror surface of the room mirror. It can be attached by a fixture or attachment. On the display unit 10a, an image that is opposite to the image captured by the imaging unit 2 provided outside the vehicle, that is, outside the passenger compartment, that is, a mirror image is displayed. The display unit 10a can be configured as, for example, an LCD (liquid crystal display), an OELD (organic electro-luminescent display), a projector device, or the like. ECU11 may be accommodated in the housing | casing provided in the place different from the display part 10a, and may be accommodated in the housing | casing of the display apparatus 10. FIG. A half mirror (not shown) may be provided on the surface side of the display unit 10a. In this case, in a state where the image display system 100 is not used and no image is displayed on the display unit 10a, the half mirror can be used as a room mirror. Moreover, the part which covers the area | region where the display part 10a does not display the image partially, for example, the area | region blacked out among half mirrors, can be used as a room mirror.

  As shown in FIG. 3, the electrical components included in the image display system 100 are connected to each other electrically or communicably via the in-vehicle network 23, for example. The electrical components include, for example, ECU 11, non-contact measuring device 13, rudder angle sensor 14, GPS 16 (global positioning system), wheel speed sensor 17, brake sensor 18a of brake system 18, accelerator sensor 19, and torque sensor of steering system 20. 20a, actuator 20b, shift sensor 21, direction indicator 22, operation input unit 24b of monitor device 24, and the like. The in-vehicle network 23 is, for example, a CAN (controller area network). Each electrical component may be electrically or communicably connected via other than CAN.

  The non-contact measuring device 13 is, for example, a sonar or radar that emits ultrasonic waves or radio waves and catches the reflected waves, and a plurality of non-contact measuring devices 13 are arranged on the body surface of the vehicle body 1. The ECU 11 measures the presence or absence of obstacles (objects) such as other vehicles and pedestrians and the distance (relative distance) to the obstacles based on the detection result of the non-contact measuring device 13. be able to. In the present embodiment, the non-contact measuring device 13 measures the distance from the rear part of the vehicle body 1 to the obstacle, and sends the measured distance to the ECU 11.

  The steering angle sensor 14 is a sensor that detects a steering amount of a steering wheel (not shown) as a steering unit, and is configured using, for example, a Hall element. Note that the steering amount is detected as, for example, a rotation angle. Depending on the steering amount provided from the rudder angle sensor 14, the display direction behind the vehicle body 1 by a panoramic image 42 to be described later may be determined.

  The wheel speed sensor 17 is a sensor that detects the amount of rotation of the wheel of the vehicle body 1 and the number of rotations per unit time, and is configured using, for example, a Hall element. The ECU 11 can calculate the amount of movement of the vehicle body 1, the vehicle speed, and the like based on the data acquired from the wheel speed sensor 17. The wheel speed sensor 17 may be provided in the brake system 18.

  The brake system 18 includes an anti-lock brake system (ABS) that suppresses the locking of the brake, a skid prevention device (ESC: electronic stability control) that suppresses the skidding of the vehicle body 1 during cornering, and an electric brake system that increases the braking force. BBW (brake by wire). The brake system 18 applies braking force to the wheels via an actuator (not shown) to decelerate or stop the vehicle body 1. The brake sensor 18a is, for example, a sensor that detects an operation amount of a brake pedal.

  The accelerator sensor 19 is a sensor that detects an operation amount of an accelerator pedal. The steering system 20 is, for example, an electric power steering system, an SBW (steer by wire) system, or the like. The torque sensor 20a detects the torque that the driver gives to the steering wheel. Further, the steering system 20 steers the front wheels by adding torque (assist torque) to the steering wheel by the actuator 20b to supplement the steering force.

  The shift sensor 21 is, for example, a sensor that detects the position of the movable part of the speed change operation part, and is configured using a displacement sensor or the like. The movable part is, for example, a lever, an arm, a button, or the like. Note that the configuration, arrangement, electrical connection form, and the like of the various sensors and actuators described above are examples, and can be variously set or changed. The direction indicator 22 outputs a signal for instructing to turn on, turn off, blink, etc. the direction indication light. This output signal can be used to estimate the traveling direction of the vehicle body 1 used when changing the display range of a panoramic image 42 described later.

  The display unit 10a of the display device 10 can be covered with a transparent operation input unit 10b. The operation input unit 10b is, for example, a touch panel. A user or the like can visually recognize an image displayed on the display screen of the display unit 10a via the operation input unit 10b. Further, the user or the like operates the image input system 100 by touching, pushing, or moving the operation input unit 10b with a finger or the like at a position corresponding to the image displayed on the display screen of the display unit 10a. Various operation inputs can be executed. The operation input unit 10b may be provided at a position other than the display unit 10a. For example, it may be provided on a steering wheel or a dashboard. In this case, the operation input unit 10b can be configured as, for example, a push button, a switch, a knob, or the like.

  Further, a monitor device 24 including a display unit 24 a different from the display unit 10 a can be provided in the vehicle body 1. The monitor device 24 is provided with a display unit 24a and an audio output device 24c. The display unit 24a is, for example, an LCD or an OELD. The audio output device 24c is, for example, a speaker. Further, the display unit 24a is covered with a transparent operation input unit 24b. The operation input unit 24b is, for example, a touch panel. A user or the like can visually recognize an image displayed on the display screen of the display unit 24a via the operation input unit 24b. The user or the like executes an operation input by touching, pushing, or moving the operation input unit 24b with a finger or the like at a position corresponding to the image displayed on the display screen of the display unit 24a. be able to. And the monitor apparatus 24 can be located in the vehicle width direction of a dashboard, ie, the center part of the left-right direction, for example. The monitor device 24 can include an operation input unit (not shown) such as a switch, a dial, a joystick, and a push button, for example. The monitor device 24 can also be used as a navigation system or an audio system. The ECU 11 can cause the display unit 24a of the monitor device 24 to display an image similar to that of the display unit 10a.

  The ECU 11 includes, for example, a CPU 11a (central processing unit), a ROM 11b (read only memory), a RAM 11c (random access memory), an SSD 11d (solid state drive), a display control unit 11e, an audio control unit 11f, and the like. The SSD 11d may be a flash memory. The CPU 11a can execute various calculations. The CPU 11a can read a program installed and stored in a nonvolatile storage device such as the ROM 11b or the SSD 11d, and execute arithmetic processing according to the program. The RAM 11c temporarily stores various types of data used in computations by the CPU 11a. The SSD 11d is a rewritable nonvolatile storage unit, and can store data even when the ECU 11 is powered off. The display control unit 11e mainly performs image display processing using image data obtained by the imaging unit 2 and image data displayed on the display unit 10a and the display unit 24a. Processing or the like can be executed. In addition, the voice control unit 11 f can mainly execute processing of voice data output from the voice output device 24 c among the calculation processes in the ECU 11. Note that the CPU 11a, the ROM 11b, the RAM 11c, and the like can be integrated in the same package. Further, the ECU 11 may have a configuration in which another logical operation processor such as a DSP (digital signal processor) or a logic circuit is used instead of the CPU 11a. Further, an HDD (hard disk drive) may be provided instead of the SSD 11d, and the SSD 11d and the HDD may be provided separately from the ECU 11.

  As shown in FIG. 4, the CPU 11 a includes various modules that are realized by reading a program installed and stored in a storage device such as the ROM 11 b and executing the program. The CPU 11a includes, for example, an image acquisition unit 30, an obstacle detection unit 32, an image synthesis unit 34, an alarm acquisition unit 36, a homography calculation unit 40, and a conversion unit 41 as modules related to rear image display control in this embodiment. Etc.

  The image acquisition unit 30 (acquisition unit) is provided via the display control unit 11e in order to create a panoramic composite image (panoramic image 42) with the rear part 1a (rear end part) of the vehicle body 1 as a viewpoint. The rear image captured by the imaging unit 2 is acquired. For example, a first rear image obtained by imaging the rear from the rear portion of the vehicle body 1 provided from the imaging unit 2a is acquired. Moreover, the 2nd back image which imaged the right back containing the right side (for example, right side body 1b) of the vehicle body 1 provided from the imaging part 2b is acquired. Further, a third rear image obtained by imaging the left rear including the left side (for example, the left side body 1c) of the vehicle body 1 provided from the imaging unit 2c is acquired. The display control unit 11e can also display the rear image provided from the imaging unit 2 on the display unit 10a and the display unit 24a as they are.

  The obstacle detection unit 32 (detection unit) detects the presence or absence of an obstacle around the vehicle body 1. Specifically, the obstacle detection unit 32 includes a first rear image captured by the imaging unit 2a, a second rear image captured by the imaging unit 2b, and a third rear image captured by the imaging unit 2c. The presence or absence of an obstacle is detected using an image processing technique such as a detection result of the non-contact measuring device 13 provided via the in-vehicle network 23 or pattern matching. The obstacle detection unit 32 can further detect the presence or absence of an obstacle in the peripheral area outside the right end and the left end of the panoramic image 42.

  The conversion unit 41 cuts out an image region suitable for creating the panoramic image 42 from the first back image, the second back image, and the third back image (this is referred to as a cut-out image), and further homogenizes them. Performs image conversion using graphics. Here, when an obstacle is detected by the obstacle detection unit 32, the conversion unit 41 uses the homography, and the shorter the distance to the obstacle detected by the non-contact measurement device 13, the first rear The cutout images of the first back image, the second back image, and the third back image are converted so that the angle of view of the image becomes wide.

  The SSD 11d stores a homography table in which homography (that is, a homography matrix) for each of the imaging units 2a, 2b, and 2c is registered for each distance. FIG. 5 is a diagram illustrating an example of a homography table according to the first embodiment. As shown in FIG. 5, in the homography table, homography for each of the imaging units 2a, 2b, and 2c is registered in advance for each distance. In FIG. 5, the homography RR is a homography for the imaging unit 2a, the homography SR is a homography for the imaging unit 2b, and the homography SL is a homography for the imaging unit 2c. The shorter the is, the larger the angle of view of the first rear image is calculated. In the example of FIG. 5, homography for each distance of 5 m is registered, but the distance is not limited to this.

  The homography calculation unit 40 calculates the homography and registers it in the homography table. The homography calculation unit 40 calculates the homography for each of the imaging units 2a, 2b, and 2c in advance using the calibration index pattern.

  FIG. 6 is a diagram illustrating an example of a calibration index pattern according to the first embodiment. In the present embodiment, the calibration index pattern 411 is a pattern in which four checkered calibration indices 412 are arranged in two stages. Here, the coordinates of the calibration index are the coordinates of the center position 413 of the calibration index.

  In the present embodiment, the first rear image captured by the imaging unit 2a of the rear portion 1a of the vehicle body 1 and the second rear image obtained by imaging the right rear including the right side of the vehicle body 1 are combined. In order to generate a panoramic image by combining the one rear image and the third rear image obtained by imaging the left rear including the left side of the vehicle body 1, the first rear image and the second rear image are partially The first rear image and the third rear image partially overlap. Therefore, the calibration index 412 is formed at a position close to the end portions on both sides of the calibration index pattern 411 so as to be imaged in the two overlapping areas.

  In the present embodiment, four calibration indices 412 are arranged on both end sides of the calibration index pattern 411, respectively. If the optical axis of each imaging unit 2 is precisely adjusted for image synthesis, even one calibration index 412 is sufficient, but in reality such precise axis adjustment has been made. Often not. For this reason, in this embodiment, precise optical axis adjustment is not performed for each of the imaging units 2 by disposing a plurality of calibration indexes 412 on both ends of the calibration index pattern 411, respectively. Even in this case, the first back image, the second back image, and the third back image can be accurately combined.

  The calibration index is not limited to this, and an arbitrary pattern can be used as the calibration index. Further, the number of calibration indices constituting the calibration index pattern is not limited to eight, and may be one or more.

  Usually, the homography calculation is performed as follows. First, the operator images the calibration index pattern 411 installed on the wall surface extending in the vertical direction behind the vehicle body 1 by each of the imaging units 2a, 2b, and 2c attached to the vehicle body 1. Here, the imaging unit 2a performs imaging so that the entire calibration index pattern 411 is included. In the imaging unit 2b, imaging is performed so that the left four calibration indices 412 of the calibration index pattern 411 are included. The imaging unit 2 c captures an image so that the four calibration indices 412 on the right side of the calibration index pattern 411 are included.

  Then, the homography calculation unit 40 obtains the coordinates of each calibration index in the captured images by the imaging units 2a, 2b, and 2c, and calculates each calibration coordinate from both the calibration index positions in the calibration index pattern 411 installed on the wall surface. Homography, which is a matrix for converting between the two, is calculated. As a homography calculation method, for example, a known method disclosed in JP-A-2006-148745 is used. In the present embodiment, multistage homography is calculated while changing the distance between the calibration index pattern and the rear part of the vehicle body 1.

  FIG. 7 is a diagram for explaining an example of the homography calculation method according to the first embodiment. In the present embodiment, as shown in FIG. 7, the installation position of the calibration indicator pattern 411 is changed so that the distance from the rear portion 1a of the vehicle body 1 changes in a plurality of stages. Then, the homography calculation unit 40 calculates the homography of each of the imaging units 2a, 2b, and 2c at each stage by the above method, and the calculated homography from the rear part 1a to the calibration index pattern 411 at each stage. It is registered in the homography table in association with the distance. Thereby, the homography table shown in FIG. 5 is generated.

  The conversion unit 41 acquires a homography corresponding to the distance to the obstacle detected by the non-contact measurement device 13 from the homography table, and uses the acquired homography to obtain the first rear image and the second rear image. Each cut-out image of the rear image and the third rear image is converted.

  The image composition unit 34 synthesizes the cut-out images of the first rear image, the second rear image, and the third rear image, which are converted by the conversion unit 41, so that the rear portion 1a (rear end) of the vehicle body 1 is obtained. A composite image (wide-angle image, panoramic image) in a panoramic format with a viewpoint of (part).

  In order to make the user more easily recognize the presence of an obstacle when the obstacle detection unit 32 detects the obstacle, the alarm acquisition unit 36, for example, an alarm displayed on the display unit 10a, for example, A character, an icon, or the like indicating an alarm mark or an obstacle is read from a storage unit such as the SSD 11d. Note that the alarm acquisition unit 36 may acquire not only a visual alarm but also audio data for performing an audio alarm, for example. For example, an alarm message such as “There is a motorcycle diagonally right behind” may be acquired and provided to the voice control unit 11f.

  Next, creation of the panoramic image 42 will be described. As described above, the imaging unit 2 includes a wide-angle lens or a fisheye lens. Therefore, it has a wide imaging range as shown in FIG. However, since an image captured with a wide-angle lens or a fish-eye lens is distorted (curved) at the periphery, when the image composition unit 34 creates a panoramic image 42, a portion with less distortion is cut out and used. For example, as shown in FIG. 9, the first rear image captured by the image capturing unit 2 a attached to the rear portion 1 a of the vehicle body 1 is composed of a substantially central portion with relatively little distortion as a first output image 52 for synthesis. To do. Moreover, the right side body 1b is reflected in the second rear image picked up by the image pickup unit 2b attached to the right door mirror 4a of the vehicle body 1, and a portion including a region on the right outer side from the first output image 52 is included. A second cut-out image 50a for synthesis is assumed. Similarly, the left side body 1c is reflected in the third rear image captured by the imaging unit 2c attached to the left door mirror 4b of the vehicle body 1 and includes a region on the left outer side from the first output image 52. Is a third cut-out image 50b for synthesis.

  Here, since the panoramic image 42 using a plurality of images can be created by a known method, a detailed description thereof is omitted. For example, a first cut-out image 52, a second cut-out image 50a, and a third cut-out image 50b are included. As shown in FIG. 8, the cutout range is determined so that the end regions of adjacent cutout images overlap. Here, it is assumed that the panoramic image 42 includes image ranges 42a, 42b, and 42c, as shown in FIG. Then, various image processing such as rotation processing, viewpoint conversion processing, homography conversion by the conversion unit 41, and the like are performed on each of the cut-out images, and the overlapping portions of the cut-out images are connected as overlap allowances. It is possible to create a panoramic image 42 with smooth continuity. In the case of the present embodiment, an example is shown in which each cut-out image is used as a layer and the panoramic image 42 is created by overlapping each other.

  In the case of FIG. 8, for example, consider a case where the first cut-out image 52 is the first layer, the second cut-out image 50a is the second layer, and the third cut-out image 50b is the third layer. The all-first-out image 52 is obtained by cutting out an area near the center of the wide-angle lens or fish-eye lens at an angle of view θ6. That is, the first output image 52 is an image with little distortion. On the other hand, the second cut-out image 50a includes the right side body 1b having a large distortion at the periphery of the imaging range. For example, the region indicated by the angle of view θ7 approaches the center of the wide-angle lens or the fisheye lens, so that the distortion is reduced. Similarly, the third cut-out image 50b includes the left side body 1c having a large distortion at the periphery of the imaging range. For example, the region indicated by the angle of view θ8 approaches the center of the wide-angle lens or the fish-eye lens, so that the distortion is reduced. FIG. 9 is an example of the rear image 48. In the first display image 52, a succeeding vehicle 56 traveling behind the vehicle body 1 (own vehicle) is shown. The second cutout image 50a shows the right side body 1b of the vehicle body 1 and the following vehicle 56. The left side body 1c of the vehicle body 1 is shown in the third cutout image 50b.

  When creating the panorama image 42, the first cut-out image 52 with little distortion is used as the uppermost layer (upper layer) when the panorama image 42 is synthesized, and the second cut-out image 50a and the third cut-out image 50b including distortion are used as the lower layer. Overlapping as. In other words, by making the regions with large distortion on the side of the first cutout image 52 side of the second cutout image 50 a and the side of the third cutout image 50 b on the side of the first cutout image 52, the lower layer of the first cutout image 52. As a whole, a panoramic image 42 with less distortion can be created as shown in FIG.

  Here, as described above, in the present embodiment, the shorter the distance between the rear part 1a of the vehicle body 1 and the obstacle, the different the homography is used for each distance so that the angle of view of the first rear image becomes wider. The cutout image is converted. That is, as shown in FIG. 7, the calibration index pattern 411 is arranged in multiple stages so that the distance from the rear part 1a of the vehicle body 1 is different, and the homography is calculated for each distance of each stage. Hereinafter, this reason will be described.

  In FIG. 7, the angle of view of the first rear image is shown for each position of the calibration index pattern 411, that is, for each distance to the calibration index pattern 411 of the rear portion 1a of the vehicle body 1. As shown in FIG. 7, it can be seen that the angle of view of the first rear image becomes wider as the distance from the rear portion 1a of the vehicle body 1 to the calibration index pattern 411 becomes shorter. When it is far from the rear portion 1a of the vehicle body 1, the angle of view is narrowed, but many obstacles are imaged. On the other hand, when the vehicle body 1 is close to the rear portion 1a, it is difficult to image the entire nearby obstacle unless the angle of view is wide.

  When calculating the homography corresponding to each of the imaging units 2a, 2b, 2c with only one position of the calibration index pattern 411, that is, without changing the distance between the rear part 1a of the vehicle body 1 and the calibration index pattern The angle of view of each of the first rear image, the second rear image, and the third rear image is only one. For example, in FIG. 7, the calibration index pattern 411 is installed on the wall surface at the position P1, the calibration index pattern 411 is imaged by the imaging units 2a, 2b, and 2c, the homography is calculated, and the calibration index pattern at other positions. When the calculation of the homography using 411 is not performed, the angle of view of the first rear image, the second rear image, and the third rear image is fixed to the angle of view when the calibration index pattern is set in P1. The

  When the captured image is converted by using the homography calculated at only one position of the calibration index pattern 411, as shown in the examples of FIGS. When there is an obstacle such as a car, a distant obstacle appears in the panoramic image. However, when there is an obstacle such as another vehicle at a distance close to the rear portion 1a of the vehicle body 1, the obstacle does not completely fall within the range of the fixed angle of view. Things may appear broken up.

  Here, since the imaging part 2a of the rear part 1a of the vehicle body 1 has a wide-angle or fish-eye lens, the entire obstacle at a nearby position is imaged including the peripheral part of the first rear image. Often. However, since the peripheral portion of the first rear image captured with a wide-angle or fish-eye lens has a lot of distortion, it cannot be used for composition for generating a panoramic image. For this reason, as described above, A range obtained by removing the peripheral portion from the rear image, that is, a range with less distortion, is cut out to synthesize the cut-out image. For this reason, the whole obstacle may not appear in the cutout image.

  FIG. 11 is a diagram illustrating an example of a cut-out image cut out from a conventional rear image. FIG. 12 is a diagram illustrating an example of a panoramic image synthesized from a conventional rear image. Since the angle of view of the first rear image is fixed regardless of the distance between the vehicle body 1 and the obstacle (following vehicle) 56, as shown in FIG. A part of the obstacle 56 such as a car may be cut off from the cutout image 52 of the first rear image. Therefore, a panorama that is converted and generated from the cut-out image 52 of the first back image, the cut-out image 50b of the second back image, and the cut-out image 50a of the third back image using homography irrelevant to the distance. In the image, as shown in FIG. 12, the nearby obstacle 56 appears partially lacking, and the joint between the cut-out image of the first rear image and the cut-out image of the second rear image is not smooth.

  For this reason, in this embodiment, as shown in FIG. 7, the distance between the calibration index pattern 411 and the rear part 1a of the vehicle body 1 is changed in a plurality of stages, and the homography calculation unit 40 performs homography for each distance of each stage. Is determined in advance. Thereby, as the distance between the calibration index pattern 411 and the rear portion 1a of the vehicle body 1 becomes shorter, a homography that can be converted into an image with a wider angle of view is required. In the present embodiment, the conversion unit 41 uses the homography corresponding to the relative distance from the rear part 1a of the vehicle body 1 to the obstacle, that is, switches the holography to convert the captured image, and the image composition unit 34 converts the panoramic image. Is generated.

  FIG. 13 is a diagram for explaining an example of homography switching according to the first embodiment. As shown in FIG. 13, when the succeeding vehicle is at the position of reference numeral 1302, the converting unit 41 uses the homography corresponding to the distance to the position P2 to extract the first rear image, the second image The clipped image of the rear image and the clipped image of the third rear image are converted. When the succeeding vehicle is at a close position such as the position indicated by reference numeral 1301, the conversion unit 41 uses the homography corresponding to the distance to the position P1 to extract the first rear image, the second image The clipped image of the rear image and the clipped image of the third rear image are converted. Thereby, the angle of view of the first rear image becomes wider than the angle of view at the position P2.

  FIG. 14 is a diagram illustrating an example of a cutout image cut out from the rear image according to the first embodiment. FIG. 15 is a diagram illustrating an example of a panoramic image synthesized from the rear image according to the first embodiment. As described above, since the first rear image is converted by homography so that the angle of view of the first rear image is wide, the entire following vehicle as the obstacle 56 appears in the first rear image as shown in FIG. As a result, as shown in FIG. 15, the entire following vehicle 56 appears without interruption in the panoramic image, and the joint between the first rear image, the second rear image, and the third rear image is smooth. It will be something.

  Next, a series of flows of the panoramic image generation process of the present embodiment configured as described above will be described. FIG. 16 is a flowchart illustrating an example of a procedure of panorama image generation processing according to the first embodiment. Note that the processing of the flowchart of FIG. 16 is repeatedly executed at a predetermined control cycle.

  First, for example, when the power switch of the vehicle body 1 is in the ON state, the imaging unit 2 captures image data, that is, moving image data at a predetermined frame rate and provides the image data to the display control unit 11e. For example, a wide-angle captured image before clipping as shown in FIG. 1 is provided. And the image acquisition part 30 acquires the captured image (1st back image, 2nd back image, 3rd back image) provided sequentially (S11). Subsequently, the obstacle detection unit 32 detects the presence or absence of an obstacle in the first rear image, the second rear image, and the third rear image (S12).

  When an obstacle is detected (S12: Yes), the non-contact measuring device 13 calculates a distance (relative distance) to the obstacle (S13). And the conversion part 41 acquires the homography corresponding to the measured distance from a homography table (S14).

  In S12, when no obstacle is detected (S12: No), the conversion unit 41 acquires standard homography (S18). Here, the standard homography can be arbitrarily determined. For example, a homography corresponding to a distance of 5 m, a homography corresponding to the farthest distance, or the like can be set as a standard homography, but is not limited thereto. Is not to be done.

  Then, the converting unit 41 cuts out each of the captured images, that is, the first back image, the second back image, and the third back image as described above, and uses each cut-out image by the homography acquired in S14. Conversion is performed (S15).

  Next, the image composition unit 34 synthesizes the cutout images converted in S15 to generate a panoramic image (S16). At this time, the image composition unit 34 performs image composition by setting the cut-out image of the first rear image as the highest layer. The display control unit 11e displays the panoramic image generated in S17 on the display unit 10a (S17).

  As described above, in this embodiment, the distance between the calibration index pattern 411 and the rear part 1a of the vehicle body 1 is changed in a plurality of stages, and homography for each distance in each stage is obtained in advance. In the present embodiment, the conversion unit 41 uses the homography corresponding to the relative distance from the rear part 1a of the vehicle body 1 to the obstacle, that is, switches the holography to convert the captured image, and the image composition unit 34 converts the panoramic image. Is generated. For this reason, the closer the obstacle is to the vehicle body 1, the wider the angle of view of the first rear image, and smoothes the joint between the first rear image, the second rear image, and the third rear image, The entire object can be displayed in a panoramic image.

  In addition, according to the present embodiment, the homography is obtained in advance for each distance obtained by changing the distance between the calibration index pattern 411 and the rear portion 1a of the vehicle body 1 in a plurality of stages, and the homography is acquired during traveling. To convert the image. For this reason, according to the present embodiment, it is possible to reduce the load of the composite image generation process while the vehicle is traveling.

(Embodiment 2)
In the first embodiment, the homography corresponding to the distance is calculated in advance and registered in the homography table in advance. In the second embodiment, the homography is calculated while the vehicle is running to generate a panoramic image. .

  The arrangement location and imaging range of the imaging unit 2 in the vehicle of the present embodiment, the configuration of the image display system, and the configuration of the image display control unit realized in the ECU of the image display control device are the same as those of the first embodiment.

  The homography calculation unit 40 of the present embodiment, when an obstacle is detected while the vehicle is running, overlaps the first rear image and the second rear image, the first rear image, and the first rear image. A feature point as a calibration index is assigned to an overlapping area with the third rear image, and homography is calculated based on the calibration index and the position of the imaging unit 2. Since the homography calculation unit 40 calculates the homography using the actual captured image while traveling, the homography for conversion so that the angle of view of the first rear image becomes wider as the distance is shorter is calculated. It will be.

  The conversion unit 41 converts each cut-out image of the first rear image, the second rear image, and the third rear image using the homography calculated by the homography calculation unit 40.

  Functions and configurations of the image acquisition unit 30, the obstacle detection unit 32, the image synthesis unit 34, the alarm acquisition unit 36, and the display control unit 11e are the same as those in the first embodiment.

  Next, a series of flows of the panoramic image generation process of the present embodiment configured as described above will be described. FIG. 17 is a flowchart illustrating an example of a procedure of panoramic image generation processing according to the second embodiment. Note that the processing of the flowchart of FIG. 17 is repeatedly executed at a predetermined control cycle.

  S11 and S12 are performed in the same manner as in the embodiment. When an obstacle is detected in S12 (S12: Yes), the homography calculator 40 overlaps the first rear image and the second rear image, and the first rear image and the third rear image. A feature point as a calibration index is assigned to the overlapping region (S32). And the homography calculation part 40 calculates homography (S32).

  If no obstacle is detected in S12 (S12: No), a predetermined standard homography is acquired in the same manner as in the first embodiment (S18). Note that even when no obstacle is detected, the homography calculation unit 40 may be configured to perform the homography calculation by adding feature points to the captured image. The steps after S15 are the same as those in the first embodiment.

  As described above, in the present embodiment, the homography calculation unit 40, when an obstacle is detected during traveling of the vehicle, the overlapping region between the first rear image and the second rear image and the first A feature point as a calibration index is assigned to an overlapping area between the rear image and the third rear image, and homography is calculated based on the calibration index and the position of the imaging unit 2. For this reason, in the present embodiment, even during traveling, the homography for conversion so that the angle of view of the first rear image becomes wider as the distance is shorter is calculated. For this reason, while the vehicle is traveling, the distance to the obstacle such as the following vehicle constantly changes, but in this embodiment, the homography corresponding to the distance is calculated and the captured image is converted during traveling. Therefore, even when the distance to the obstacle changes while the vehicle is running, the joint between the first rear image, the second rear image, and the third rear image is smoothed to more reliably Can be displayed in a panoramic image.

(Embodiment 3)
In the third embodiment, when a plurality of obstacles are detected while the vehicle is running, a panoramic image corresponding to the distance from each obstacle is generated and displayed in a switched manner.

  The arrangement location and imaging range of the imaging unit 2 in the vehicle of the present embodiment, the configuration of the image display system, and the configuration of the image display control unit realized in the ECU of the image display control device are the same as those of the first embodiment.

  When a plurality of obstacles are detected, the conversion unit 41 according to the present embodiment acquires a homography corresponding to the distance to the obstacle for each of the detected obstacles, and obtains the acquired plurality of homography. The first rear image, the second rear image, and each clipped image of the third rear image are converted.

  The image composition unit 34 generates a panoramic image from each clipped image for each of a plurality of obstacles. The display control unit 11e performs control for switching the panoramic image for each of the plurality of harmful objects and displaying the panoramic image on the display unit 10a.

  Next, a series of flows of the panoramic image generation process of the present embodiment configured as described above will be described. FIG. 18 is a flowchart illustrating an example of a procedure of panorama image generation processing according to the third embodiment. Note that the processing of the flowchart of FIG. 18 is repeatedly executed at a predetermined control cycle.

  S11 and S12 are performed in the same manner as in the embodiment. When an obstacle is detected in S12 (S12: Yes), the non-contact measuring device 13 measures the distance from the vehicle body 1 for each obstacle (S51). And the conversion part 41 acquires the homography according to the distance to each obstacle for every obstacle from a homography table (S52). Thereby, a plurality of homography according to the distance to the obstacle is acquired.

  If no obstacle is detected in S12 (S12: No), a predetermined standard homography is acquired in the same manner as in the first embodiment (S18).

  Then, the conversion unit 41 cuts out each cut-out image for each obstacle as described above from each of the captured images, that is, the first back image, the second back image, and the third back image. Conversion is performed by each acquired homography (S55).

  Next, the image composition unit 34 synthesizes the cut-out images converted in S55 for each obstacle to generate a panoramic image (S56). Thereby, a plurality of panoramic images are generated. The display control unit 11e displays the plurality of panoramic images generated in S56 on the display unit 10a while switching (S57). The display control unit 11e performs switching display by applying an effect such as cross fading when switching the display of a plurality of panoramic images.

  As described above, in the present embodiment, when a plurality of obstacles are detected, a panoramic image corresponding to the distance from each obstacle is generated and displayed in a switched manner. That is, in this embodiment, the homography corresponding to the distance to the obstacle is obtained for each of the plurality of obstacles detected by the conversion unit 41, and the first rear image is obtained using the obtained plurality of homography. The second rear image and the third rear image, and the image composition unit 34 generates a panoramic image from each cut image for each of the plurality of obstacles, and the display control unit 11e The panoramic image is switched for each of the plurality of harmful objects and displayed on the display unit 10a. Therefore, according to the present embodiment, even when a plurality of obstacles are detected during traveling, a plurality of panoramic images in which the first rear images having the angle of view corresponding to the distance to each obstacle are combined. The entire obstacle can be displayed.

  In the present embodiment, when a plurality of obstacles are detected, a pre-calculated homography is acquired and image-converted to generate a plurality of panoramic images, but as in the second embodiment, The conversion unit 41 and the image composition unit 34 may be configured to calculate a homography according to the relative distance to each obstacle and generate a plurality of panoramic images at a predetermined calculation cycle during traveling. In this case, in this case, an optimal panoramic image can be displayed according to the relative distance to the obstacle that changes during traveling, and the entire obstacle can always be displayed.

  Further, in the present embodiment, when a plurality of obstacles are detected, a panoramic image is generated and switched for each of the plurality of obstacles, but even when only one obstacle is detected, The conversion unit 41 and the image synthesis unit 34 are configured to generate a plurality of panoramic images from a plurality of holography corresponding to different distances in accordance with changes in relative speed and relative distance of the images, and to switch and display the plurality of panoramic images. It may be configured. In this case, an optimal panoramic image can be displayed according to the relative speed and relative distance with the obstacle that changes during traveling, and the entire obstacle can always be displayed. In this case, the display control unit 11e can be further configured to determine the panoramic image switching timing in accordance with changes in relative speed and relative distance from the obstacle.

  In this embodiment, when a plurality of obstacles are detected, a panoramic image is generated and switched for each of the plurality of obstacles. However, even when only one obstacle is detected, it corresponds to different distances. The conversion unit 41 and the image composition unit 34 are configured to generate a plurality of panoramic images from a plurality of holography and to switch and display the plurality of panoramic images according to changes in relative speed and relative distance with the obstacle. May be. In this case, an optimal panoramic image can be displayed according to the relative speed and relative distance with the obstacle that changes during traveling, and the entire obstacle can always be displayed.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 Car body 2, 2a, 2b, 2c Image pick-up part 11e Display control part 10a, 24a Display part 13 Non-contact distance measuring device 30 Image acquisition part 32 Obstacle detection part 34 Image composition part 36 Alarm acquisition part 40 Homography calculation part 41 Conversion Part

Claims (5)

A first rear image obtained by imaging the rear from the rear of the vehicle body, a second rear image obtained by imaging the right rear including the right side of the vehicle body, and a third rear imaged by the left rear including the left side of the vehicle body An acquisition unit for acquiring an image;
A detection unit for detecting presence or absence of an obstacle in the first rear image, the second rear image, and the third rear image;
When the obstacle is detected, using the homography calculated based on the distance from the rear part of the vehicle body to the obstacle, the first rear image, the second rear image, and the third A conversion unit for converting the back image;
The first rear image, the second rear image, and the third rear image converted by the conversion unit are combined to generate a panoramic composite image with the rear end of the vehicle body as a viewpoint. An image composition unit;
A control unit that performs control to display the composite image on a display device;
An image display control device. The conversion unit converts the first back image, the second back image, and the third back image so that the angle of view of the first back image becomes wider as the distance is shorter. ,
The image display control apparatus according to claim 1. A storage unit that stores a homography table in which the angle of view of the first rear image becomes wider as the distance is shorter, and the distance is associated with the homography table;
A distance measuring unit that measures the distance from the rear part of the vehicle body to the obstacle, and
The conversion unit acquires a homography corresponding to the distance measured by the distance measurement unit from the homography table, and uses the acquired homography to obtain the first rear image and the second rear image. And the third back image,
The image display control device according to claim 1. When the obstacle is detected, a calibration index is provided in an overlapping area between the first rear image and the second rear image and an overlapping area between the first rear image and the third rear image. A homography calculation unit that calculates the homography based on the calibration index and the position of the imaging unit,
The conversion unit converts the first back image, the second back image, and the third back image using the homography calculated by the homography calculation unit.
The image display control device according to claim 1. When the plurality of obstacles are detected, the conversion unit acquires a homography corresponding to the distance for each of the detected obstacles, and uses the acquired homography to obtain the first rear image. And the second rear image and the third rear image,
The image composition unit generates the composite image for each of the plurality of obstacles,
The control unit performs control for switching the composite image for each of the plurality of obstacles and displaying the composite image on the display device.
The image display control apparatus according to claim 3.

Images