JP7291900B2 - Visibility support image generation device and image conversion program - Google Patents

Description

TECHNICAL FIELD The present invention relates to an apparatus for generating a visibility support image to be displayed to an occupant of a vehicle, and an image conversion program.

2. Description of the Related Art Some vehicle door mirrors use aspherical mirrors, which are optical mirrors, in order to reduce blind spots in the lateral direction (left and right directions as seen by the driver of the vehicle, away from the own vehicle). FIG. 1 is a schematic diagram of an existing optical aspherical mirror, in which a door mirror M has a normal mirror portion M1 and an aspherical portion M2. The aspherical portion M2 changes its curvature so that a wider range of images can be seen in the lateral direction.

Also in a CMS (camera monitoring system) in which the door mirror M is further provided with a camera for imaging, the angle of view is widened in the lateral direction by the principle imitating the aspherical mirror, which is the optical mirror. That is, the area corresponding to the aspherical portion M2 of the door mirror M in the image captured by the camera is compressed in the horizontal direction, and the image after the compression is displayed on the display.

In Patent Document 1, when an image obtained by enlarging or compressing a picked-up image of the rear side of a vehicle at a magnification factor that varies in the lateral direction is displayed on a display, an image that should extend in the front-rear direction of the own vehicle is displayed. A horizontal scale that corrects the scale of each part of the captured image in the horizontal direction at a set horizontal scaling factor in order to prevent the running area division lines such as white lines from being displayed crookedly in the image displayed on the display. Correction processing and further partial vertical scale correction with a vertical scaling factor set so that the image of the driving zone division line, which appears curved in the horizontal scale corrected image, extends straight. It is disclosed to perform a vertical scale correction process that

JP 2013-85142 A

By the way, from the point of view of an occupant of a vehicle, especially a driver (hereinafter referred to as a driver, etc.), the driver, etc., sees a display image displayed on an in-vehicle monitor or the like. The original image of this display image is captured by a camera provided in a door mirror or the like. It is preferable that this display image is displayed in a state in which there is little sense of incongruity.

In addition, since the door mirror is for grasping the situation outside the vehicle (hereinafter referred to as "outside world"), it is also desired that the display image is capable of appropriately grasping the situation in the outside world.

From the above viewpoint, an object of the present disclosure is to provide a display image that gives less sense of incongruity and allows an appropriate grasp of the situation in the outside world.

A visibility support image generation device for generating a visibility support image of a vehicle includes a camera that captures an image from the vehicle and a processing unit, and the processing unit converts the captured image captured by the camera, so that the A field of vision support image is generated, and the image conversion is such that a compression rate in the horizontal direction of the captured image is higher than a compression rate in the vertical direction of the captured image, centering on a deep vanishing point included in the captured image. As described above, the captured image is compressed. With the above configuration, the generated visual field support image has less sense of incongruity and allows appropriate grasping of the situation of the outside world.

It is possible to provide a display image that gives less sense of incongruity and enables an appropriate grasp of the situation in the outside world.

Schematic diagram of an existing optical aspherical mirror. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic diagram showing the principle of image generation by the visibility support image generation device 1 of the present disclosure, (a) a diagram showing an example of a lens model, (b) a diagram showing a captured image (input image), and (c) a display image. (output image). 1 is a configuration diagram showing an embodiment of a visibility support image generation device 1 of the present disclosure; FIG. 4 is a flowchart showing an example of image processing performed by a processing unit 11; FIG. Explanatory drawing of a perfect circle lens model. FIG. 4 is a comparison diagram between the case of generating an output image I _out from an input image I _in according to the conventional technique and the case of the visibility support image generation device 1 of the present disclosure; FIG. 10 is a second comparison diagram between the conventional art and the visibility support image generation device 1 of the present disclosure when an output image I _out is generated from an input image I _in ; FIG. 11 is a third comparison diagram between the case of the prior art and the case of the visibility support image generation device 1 of the present disclosure when the output image I _out is generated from the input image I _in ; FIG. 10 is a view showing a visibility support image (output image) I _out generated using a conventional visibility support image generation device; 1 is a configuration diagram showing an embodiment of a visibility support image generation device 1B of the present disclosure; FIG. FIG. 3 is a diagram showing a sensor image captured by a camera 12 included in a visibility assistance image generation device 1B of the present disclosure and a visibility assistance image MP displayed on a display device 13; It is a figure which shows the following in the visibility support image MP of the detected vehicle display part D, (a) initial state, (b) following state, (c) following end state is shown, respectively. FIG. 10 is a comparison diagram between a conventional technique and a visibility support image generating device 1B of the present disclosure when a visibility support image is generated from an input image _Iin . It is a figure which shows 1 aspect of layout, (a) A sensor image, (b) Visibility support image MP is shown. FIG. 4 illustrates allocation based on polar coordinates; FIG. 11 is a processing flowchart based on the system configuration shown in FIG. 10; The block diagram which shows the Example of 1 C of visibility support image generation apparatuses of this indication. The figure which shows the various installation examples of a vehicle-mounted camera. FIG. 18 is a processing flow diagram based on the system configuration shown in FIG. 17 , showing (a) processing for representing a sense of distance virtually, and (b) brightness control processing for the display device 13 ; 3A and 3B are comparison diagrams of visibility support images displayed on the display device 13, (a) a diagram showing a gaze point on the display device, (b) a visibility support image when the visibility support image generation device 1C of the present disclosure is not used, (c) A view support image MP when the view support image generation device 1C of the present disclosure is used.

Below, it is assumed that the vehicle is a right-hand drive vehicle, and that a display image is displayed on a display device provided in the vehicle using a captured image captured by a camera provided on the right door mirror, which is likely to be a blind spot for the driver. A detailed description will be given with appropriate reference to the drawings. However, it is not intended to limit the claimed subject matter to only this premise. For example, a moving body other than a vehicle, the position of the steering wheel of the vehicle (left steering wheel, right steering wheel, automatic driving without a steering wheel in the first place), and the camera mounting position (left door mirror, right door mirror, etc.). Deformation is possible.

The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure and are not intended to limit the claimed subject matter thereby.

FIG. 2 is a schematic diagram showing the principle of image generation by the visibility support image generation device 1 (described later based on FIG. 3) of the present disclosure, (a) a diagram showing an example of a lens model, (b) a captured image (input image), and (c) a diagram showing a display image (output image).

In the visibility support image generation device 1 of the present disclosure, the compression rate in the horizontal direction of the captured image is higher than the compression rate in the vertical direction of the captured image, centering on the deep vanishing point included in the captured image. A geometric lens model is used that can compress the captured image. As the geometric lens model, in this embodiment, the oblong elliptical lens model E shown in FIG. 2(a) is used.

Let b be the length of the major axis (vertical axis) and c be the length of the minor axis (horizontal axis) of the oblong elliptical lens model E, then b>c. As will be described later, if this vertically elongated elliptical lens model E is used, the compression rate of the image will linearly change from the horizontal direction to the vertical direction. In other words, assuming that the angle of deflection is 0 degrees in the horizontal direction and the angle of deflection is 90 degrees in the vertical direction, the compression rate decreases linearly as the angle of deflection increases from 0 degrees to 90 degrees. Also, as will be described later, the greater the distance from the center of the ellipse, the greater the compression rate.

FIG. 2(b) is a diagram showing the input image _Iin , which corresponds to a captured image captured by a camera attached to the right door mirror of the vehicle 100 in this embodiment.

The body of the own vehicle 100 is reflected on the left side of the input image _Iin . Further, although this is merely an example for explanation, objects OBJ1 to OBJ4 are reflected in the input image _Iin . In this example, each of the objects OBJ1-OBJ4 is a vehicle different from the own vehicle 100, which is traveling in a lane adjacent to the own vehicle 100. FIG.

FIG. 2(c) is a diagram showing the output image _Iout , which in this embodiment corresponds to a display image displayed toward the driver holding the right steering wheel by a monitor mounted inside the own vehicle 100. do.

As can be seen from FIG. 2(c), the output image I _out also includes the body of the own vehicle 100 and the objects OBJ1 to OBJ4. Comparing the input image I _in (FIG. 2(b)) and the output image I _out (FIG. 2(c)), the objects OBJ1 to OBJ4 appearing in the output image I _out are horizontally compressed. compression rate is higher than the compression rate in the longitudinal direction. That is, the angle of view widens in the horizontal direction. It should be noted that such an output image I _out is a display image that gives less sense of incongruity and allows an appropriate grasp of the situation of the external world, which will be described later.

FIG. 3 is a configuration diagram showing an embodiment of the visibility support image generation device 1 of the present disclosure. A visibility support image generation device 1 of the present disclosure includes a processing unit 11 and a camera 12 . In addition, you may provide the component other than these. As illustrated, the visibility support image generation device 1 may further include a non-volatile memory 14 and the like.

The processing unit 11 is a component that performs information processing in the visibility support image generation device 1 . The processing unit 11 performs image processing, processes commands and signals input from other components within the device and the outside of the device, and conversely, processes commands and signals to other components within the device and the outside of the device. may be sent.

The camera 12 is means for capturing an image from the vehicle to obtain the above-mentioned input image _Iin . In this embodiment, the camera 12 is a camera attached to a door mirror for imaging the rear side of the vehicle, but is not limited to this. For example, it may be a camera or the like that images the front or rear of the vehicle.

The display device 13 is a device that can display the display image generated by the visibility support image generation device 1 . Typically, the display device 13 is a monitor or the like provided inside the vehicle 100, but is not limited to this. A driver or the like sees the display image displayed on the display device 13 . In FIG. 3 , the display device 13 is separate from the visibility support image generation device 1 . However, the display device 13 may be included in the visibility support image generation device 1 .

The nonvolatile memory 14 may store a program used for image processing performed by the processing unit 11, various parameter information, a conversion table based on a perfect circle lens model (described later), and the like.

Note that the components included in the visibility support image generation device 1 may be further integrated, or conversely may be further divided into a plurality of subcomponents.

FIG. 4 is a flowchart showing an example of image processing performed by the processing unit 11. As shown in FIG.

FIG. 4 shows an example of processing for creating a conversion table used in image conversion processing for generating an output image _Iout based on an input image _Iin . As a prerequisite, in this example, the oblong elliptical lens model E shown in FIG. 2(a) is used. Moreover, as will be described later with reference to FIG. 5, in this example, lens distortion is also removed.

In step S01, the processing unit 11 takes in the input image _Iin . The captured image may be held in a memory or the like (not shown).

In step S02, the processing unit 11 determines whether or not the next pixel to be processed remains. If there is a next pixel (yes in the drawing), the process proceeds to step S03. If the next pixel does not remain (no in the drawing), it indicates that the process has been completed for all of the input image _Iin , and the conversion table creation process ends.

In step S03, the processing unit 11 selects the next pixel to be processed. Let the coordinates of the selected pixel be coordinates P for convenience.

In step S04, the processing unit 11 calculates the distance Do between the central coordinate O and the coordinate P. FIG. Here, the central coordinate O means the deep vanishing point contained within the input image _Iin . For example, using FIG. 2B, a plurality of white lines W are drawn on the road surface on which the automobile is running. On the extension lines of these white lines W, there is a point where the extension lines intersect with one point. This intersection point is the deep vanishing point. In FIG. 2(b), there is a deep vanishing point (central coordinate O) inside the image on the upper left side.

In step S05, the processing unit 11 converts the distance Do using the conversion table of the perfect circle lens model, and determines the converted distance Do'. This step S05 will be described in further detail below with reference to FIG. 5 as well.

FIG. 5 shows an explanatory diagram of the perfect circle lens model. A lens 50 shown in FIG. 5 is a general circular lens used in a pinhole camera or the like. It should be noted that FIG. 5 shows a state in which the perfect circular lens is viewed not from the front but from the side.

The image position on the evaluation surface 51 of the optical system is represented by the distance from the optical axis and is called the image height. An ideal image height 53 is an ideal image height. However, the image height of a normal optical system is not the ideal image height 53 because it is affected by lens distortion and the like. On the other hand, the real image height 52 is the image height indicating the position where the image is actually formed on the evaluation plane. Note that the distance Do between the central coordinate O and the coordinate P calculated in step S04 corresponds to the actual image height 52. As shown in FIG.

Then, the real image height 52 at each coordinate and the ideal image height 53 at each coordinate are combined to form one conversion table. In the example shown in FIG. 5, lens 50 is a perfect round lens. Therefore, the conversion table obtained by combining the real image height 52 shown in FIG. 5 and the ideal image height 53 of the perfect circular lens is the conversion table of the perfect circular lens model. Using the conversion table of the perfect circular lens model, the real image height 52 affected by the lens distortion can be converted into the ideal image height 53 (of the perfect circular lens model) from which the lens distortion is removed.

The conversion table for the perfect circle lens model as described above may be stored in, for example, the nonvolatile memory 14 or the like. Then, in step S05, the processing unit 11 converts the distance Do, which is the actual image height 52, into the converted distance Do', which corresponds to the ideal image height 53 of the perfect circular lens model, using the conversion table of the perfect circular lens model. to decide.

In step S06, the processing unit 11 calculates the image height change rate a by the perfect circle lens model. Note that a=(Do'/Do)-1.

The image height change rate a is a value obtained by dividing the distance Do', which is the ideal image height 53 of the perfect circle lens model, by the distance Do, which is the actual image height 52, and subtracting 1 from the result. For example, when the distance Do'=120, which is the ideal image height 53, and the distance Do=100, which is the real image height 52, the image height change rate a=(120/100)-1=0.2. This means that the ideal image height 53 (of the perfect circular lens model) changes (increases in this case) by 20% with respect to the real image height 52 .

Here, the visibility support image generation device 1 in the present disclosure uses the vertically elongated elliptical lens model E (FIG. 2(a)), as already described. Therefore, it is necessary to mix the elements of the vertically oblong ellipse into the image height change rate a calculated in step S06. Therefore, the following processing is performed in subsequent steps S07 and subsequent steps.

In step S07, the processing unit 11 calculates the coordinates of the intersection point P1 between the straight line extending from the coordinate O to the coordinate P and the longitudinal elliptic function E defined by the major axis b and the minor axis c. In the next step S08, the processing section 11 calculates the distance D1 between the coordinate O and the coordinate P1.

The lengths of the long axis b and the short axis c can be determined as appropriate. In this example, the vertically long elliptical lens model E is used, so the long axis b extends in the vertical direction (Y-axis direction) and the short axis c extends in the horizontal direction (X-axis direction). c<b.

Here, two specific examples are shown for easier understanding.

First, a first specific example will be shown. Assume that the coordinate P exists on the X-axis with the coordinate O as the origin. That is, assume that the coordinates P=(m, 0). m is any positive real number. At this time, the coordinates of the intersection point P1 between the straight line extending from the coordinate O to the coordinate P and the vertical elliptic function E are (c/2, 0), so D1=c/2.

A second specific example is shown. Assume that the coordinate P exists on the Y-axis with the coordinate O as the origin. That is, assume that the coordinates P=(0, n). n is any positive real number. At this time, the coordinates of the intersection point P1 between the straight line extending from the coordinate O to the coordinate P and the vertical elliptic function E are (0, b/2), so D1=b/2.

Here, when the above two specific examples are compared, since c<b, c/2<b/2. That is, the value of the distance D1 is larger in the second specific example than in the first specific example.

In subsequent step S09, the processing unit 11 calculates the compression coefficient A _p =(a/D1)+1 for the coordinate P. As described above, a is the image height change rate, and D1 is the distance from the coordinate O to the coordinate P1. Since D1 is used here as the reciprocal number, the magnitude relationship of the values is reversed between the above two specific examples. That is, A _p =(2a/c)+1 in the first specific example has a larger compression coefficient A _p than A _p =(2a/b)+1 in the second specific example. This compression factor A _p corresponds to the compression ratio.

That is, the first specific example with coordinates P=(m, 0) in the horizontal direction has a higher compression factor A than the second example with coordinates P=(0, n) in the vertical direction. The value of _p is large. This indicates that the compression rate in the horizontal direction is higher than the compression rate in the vertical direction centering on the coordinate O (deep vanishing point).

In subsequent step S10, the processing unit 11 writes the compression coefficient _Ap for the coordinate P into the conversion table of the vertically elongated elliptical lens model E as the ideal image height 53 of the vertically elongated elliptical lens model. That is, the real image height 52 and the ideal image height 53 of the oblong elliptical lens model E are combined and recorded. As can be seen from the description so far, the conversion table for the oblong elliptical lens model E is a modification of the above-described conversion table for the perfectly circular lens model.

Then, the process returns to step S02. That is, the combination of the real image height 52 and the ideal image height 53 of _the oblong elliptical lens model E is is recorded, and this is used as a conversion table for the oblong elliptical lens model E.

As described above, the conversion table for the elongated elliptical lens model E can be created by transforming the conversion table for the perfect circular lens model.

By applying the conversion table of the oblong elliptical lens model E to the input image _Iin , the output image _Iout can be generated. More specifically, the line segment from the coordinate O to the coordinate P in the input image I _in is compressed by A _p times (the distance Do is changed to the distance Do/A _p ). In this compression, the compression rate in the horizontal direction around the coordinate O (deep vanishing point) is higher than the compression rate in the vertical direction.

The lengths of the major axis b and the minor axis c of the oblong elliptical lens model E can be changed as appropriate. In other words, the visibility support image generation device 1 of the present disclosure can independently adjust the horizontal compression rate centered on the coordinate O (deep vanishing point) and the vertical compression rate.

FIG. 6 is a comparison diagram between the case of generating the output image _Iout from the input image _Iin according to the prior art and the case of the visibility support image generation device 1 of the present disclosure. FIG. 6(a) is the input image _Iin . FIG. 6B shows an output image I _out generated from the input image _I in using the conventional technique. FIG. 6C shows an output image I _out generated using the visibility support image generation device 1 of the present disclosure for the same input image I _in .

In the input image _Iin shown in FIG. 6(a), the vehicle body of the own vehicle 100 is reflected on the left side thereof. Although it is an example for explanation, objects OBJ1 to OBJ4 are reflected in the input image _Iin . In this example, the objects OBJ1 to OBJ4 are vehicles different from the own vehicle 100, which are traveling in lanes adjacent to the own vehicle.

The output image _{Iout generated} by the conventional technique shown in FIG. It is generated with further directional scaling. In other words, two steps of enlargement/reduction are performed in the horizontal direction and the vertical direction. This also applies to FIGS. 7(b) and 8(b), which will be described later.

Looking at the output image _Iout shown in FIG. 6B, the shapes of the vehicles OBJ1 and OBJ2 are significantly different from the input image _Iin . These are greatly crushed in the lateral direction, making it difficult to recognize the original shape. In addition, the sense of distance in the lateral direction also fluctuates greatly, and the inter-vehicle distance from OBJ1 to OBJ2 is extremely shortened. When viewed from the viewpoint of a driver or the like, although the presence of an object itself can be perceived, it may be difficult to determine what the object really is. Also, the inter-vehicle distance between OBJ1 and OBJ2 is originally sufficiently large (FIG. 6(a)). However, in FIG. 6(b) it appears that they are likely to collide with each other.

On the other hand, in the case of the output image I _out generated by the visibility support image generation device 1 of the present disclosure, shown in FIG. Also, the change in the sense of distance in the lateral direction is moderate. Therefore, from the viewpoint of the driver or the like, it is easy to recognize the shapes of the objects OBJ1 to OBJ4 moving in the next lane. In addition, there is no confusion due to sudden changes in the sense of distance.

FIG. 7 is a second comparison diagram between the prior art and the visibility support image generation device 1 of the present disclosure when the output image _Iout is generated from the input image _Iin . FIG. 7(a) is the input image _Iin . FIG. 7B shows an output image I _out generated from the input image _I in using the conventional technique. FIG. 7C shows an output image I _out generated using the visibility support image generation device 1 of the present disclosure for the same input image I _in .

In FIG. 7, straight lines L1 and L2 orthogonal to the traveling direction of the vehicle 100 are added as auxiliary lines. A straight line L1 is a line near the host vehicle 100, and a straight line L2 is a line far from the host vehicle 100. FIG.

In the input image _Iin shown in FIG. 7A, the two straight lines L1 and L2 are parallel.

Looking at the output image _Iout generated by the conventional technique shown in FIG. 7B, the two straight lines L1 and L2 have greatly different slopes. This difference in inclination is recognized as image distortion or discomfort from the viewpoint of the driver or the like. In addition, the multiple vehicles reflected in the image are traveling in a direction orthogonal to the two straight lines L1 and L2. In other words, when the output image I _out is a moving image, the vehicle reflected in this moving image should actually be traveling straight, but as it moves to the right side of the image, it is traveling while curving. It looks like This also makes the driver or the like feel uncomfortable.

On the other hand, in the case of the output image I _out generated by the visibility support image generation device 1 of the present disclosure, shown in FIG. become. Therefore, the sense of incongruity for the driver or the like is small. In addition, when the output image I _out is a moving image, the movement of the vehicle reflected in the moving image looks more natural, so there is less sense of incongruity.

FIG. 8 is a third comparison diagram between the prior art and the visibility support image generation device 1 of the present disclosure when the output image _Iout is generated from the input image _Iin . FIG. 8(a) is the input image _Iin . FIG. 8B shows an output image I _out generated from the input image _I in using the conventional technique. FIG. 8C shows an output image I _out generated using the visibility support image generation device 1 of the present disclosure for the same input image I _in .

In FIG. 8, an arrow L3 for explanation is put in the vicinity of the front bumper of OBJ2 which is a vehicle. This arrow L3 indicates a straight line perpendicular to the traveling direction of the own vehicle 100, which is provided by the object reflected in the image.

Looking at the output image I _out generated by the conventional technique shown in FIG. 8(b), it can be seen that the arrow L3 is curved. In other words, from the viewpoint of the driver or the like, the linearly formed member (the front bumper or the like) that constitutes the object OBJ2 appears curved. This also makes the driver or the like feel uncomfortable.

On the other hand, in the case of the output image I _out generated by the visibility support image generation device 1 of the present disclosure, shown in FIG. However, it is reflected in the eyes of drivers etc. as a shape close to a straight line. That is, there is little sense of incongruity for the driver or the like.

As described above, the output image I _out generated by the visibility support image generation device 1 of the present disclosure is an image with little discomfort.

In addition to this, the output image I _out generated by the visibility support image generation device 1 of the present disclosure has a compression rate in the horizontal direction of the captured image centered on the deep vanishing point included in the captured image. The captured image is compressed so that the compression rate in the vertical direction is higher than that in the vertical direction (see FIGS. 1 to 5). Therefore, it is still possible to enjoy the advantage of the optical aspherical mirror that the angle of view is enlarged in the horizontal direction.

That is, the visibility support image generation device 1 of the present disclosure can generate an output image I _out that gives less sense of discomfort and enables an appropriate grasp of the situation of the external world.

In addition, some supplementary items will be explained below.

In the flow chart described with reference to FIG. 4, a conversion table for the oblong elliptical lens model E is created based on the conversion table for the perfect circle lens model, and the conversion table for the oblong elliptical lens model E is used to generate the output image _Iout . It was an example. By storing the conversion table of the longitudinal elliptical lens model E in the non-volatile memory 14 or the like, it becomes unnecessary to generate a new conversion table each time the input image _Iin is input. In other words, the output image _Iout can be generated by referring to the previously stored conversion table of the longitudinally elongated elliptical lens model E for the input image _Iin . Conversely, the compression coefficient A _p may be dynamically calculated as described above each time the input image I _in is input, and then the output image I _out may be generated.

Further, in the above embodiment, the processing section 11 performs the processing related to steps S01 to S10. A program related to this processing may be stored in the non-volatile memory 14 or the like, and the processing unit 11 may read out this program and perform image processing. On the other hand, the above processing may be performed by hardware processing instead of software processing. For example, the processing may be performed by a dedicated circuit or the like.

Next, a supplementary explanation of the compression rate will be given. Two specific examples are shown for step S09 in FIG. That is, the compression rate (compression coefficient A _p ) was compared in two directions, the horizontal direction (first specific example) and the vertical direction (second specific example). However, there is also a diagonal compression ratio in between. As described above, the conversion table for the oblong elliptical lens model E is a modified version of the conversion table for the perfectly circular lens model. Since the perfect circle is converted into a vertically oblong ellipse, the compression ratio (compression coefficient A _p ) linearly changes from the horizontal direction to the vertical direction (see FIG. 2(a)). In other words, when the deflection angle is 0 degrees in the horizontal direction and the deflection angle is 90 degrees in the vertical direction, the compression rate (compression coefficient A _p ) decreases linearly as the deflection angle increases.

Also, as the perfect circle lens model, a lens model in which the compression ratio increases as the distance from the center increases can be used. The longitudinally elongated elliptical lens model E created based on such a perfect circle lens model also increases in compression ratio as the distance from the center (deep vanishing point) increases.

Next, for the purpose of the present disclosure, which is to provide a display image that gives less sense of incongruity and allows an appropriate grasp of the situation in the outside world, the visibility support image generation device 1B, which is a contrivance to make the aspherical portion movable, will be described.

A driver or the like in a vehicle equipped with a CMS (camera monitoring system) sees a visual field support image on a display device provided in the vehicle. However, the visibility support image generated by using the conventional visibility support image generation device is so close to one end of the visibility support image that corresponds to the aspherical portion M2 (see FIG. 1) of the door mirror M, which is an optical mirror. strongly compressed. Then, a driver or the like who sees the visibility support image feels uncomfortable.

An example will be given using FIG. FIG. 9 is a diagram showing a view support image (output image) I _out generated using a view support image generation device of the prior art. A black arrow is added at the bottom of FIG. 9 to represent the compression rate. That is, the compression rate increases toward the right side of the view support image (output image) _Iout .

The visibility support image (output image) I _out shows a situation in which an object OBJ2, which is a vehicle, is traveling in a lane adjacent to own vehicle 100, and approaches and overtakes own vehicle 100 from behind. At this time, as the object OBJ2 moves to the right side of the visual field support image (output image) _Iout corresponding to the aspherical portion M2, the shape of the object OBJ2 is greatly crushed by the image compression. By comparing the object OBJ1 shown in FIG. 9, which is greatly crushed to the left and right, and the object OBJ2, which is not as crushed to the left and right as the object OBJ1, which is also shown in FIG.

Also, the moving speed of the object OBJ2 on the view support image (output image) I _out appears to gradually increase as the distance to the own vehicle 100 approaches. However, on the right side of the image corresponding to the aspherical portion M2, the image is compressed, so the object OBJ2 appears to decelerate rapidly. Then, in the eyes of the driver of the own vehicle 100, it appears as if the object OBJ2, which is a vehicle, gradually accelerates and approaches the own vehicle 100, and then suddenly decelerates and overtakes the own vehicle 100.例文帳に追加Such image display gives a great sense of discomfort.

Therefore, in the visibility support image generation device 1B of the present disclosure, the above-described discomfort is suppressed in the optical aspherical mirror image display by the CMS (camera monitoring system). In order to achieve this, the method of compressing the display portion corresponding to the aspherical portion of the optical aspherical mirror is changed according to the display position of the vehicle.

More specifically, the highly compressed portion in the visibility support image is set to a location other than the detected vehicle DC (described later) displayed in the visibility support image. Then, the compression strength at the position where the detected vehicle DC is displayed is prevented from changing abruptly.

The visibility assistance image generating device 1B of the present disclosure as described above can provide a visibility assistance image that does not cause discomfort to the driver or the like. Furthermore, since the visibility support image still has a portion corresponding to the aspherical portion, it is possible to display with a wide angle of view. Hereinafter, the view support image generation device 1B of the present disclosure will be described in detail.

FIG. 10 is a configuration diagram showing an embodiment of the visibility support image generation device 1B of the present disclosure. Each device shown in FIG. 10 is mounted on the own vehicle 100 . The hardware configuration shown in FIG. 10 is basically the same as the visibility support image generation device 1 described based on FIG. The embodiment shown in FIG. 10 differs from the embodiment shown in FIG. 3 in that the visibility support image generation device 1B is connected to the approaching vehicle position calculation device 15 .

The approaching vehicle position calculation device 15 may be, for example, a millimeter wave radar installed inside the rear bumper of the vehicle or the like, and can detect a vehicle traveling in an adjacent lane and calculate the position of the vehicle. However, other distance measuring devices such as TOF sensors may be used to detect vehicles traveling in adjacent lanes.

Also, the approaching vehicle position calculation device 15 may be the camera 12, for example. That is, the camera 12 captures an image of the outside world (particularly, the adjacent lane) viewed from the host vehicle 100, and from the captured image, the position of the vehicle traveling in the adjacent lane may be obtained by software processing.

What is common to all of the above cases is that the position of the closest vehicle (hereinafter referred to as the detected vehicle DC) traveling in the adjacent lane of the own vehicle 100 is calculated by the adjacent vehicle position calculation device 15. be. Such a vehicle has the highest risk of colliding with own vehicle 100 . The visibility support image generation device 1B can perform image processing, which will be described later, by acquiring the position information of the detected vehicle DC from the proximity vehicle position calculation device 15 .

FIG. 11 is a diagram showing a sensor image captured by the camera 12 included in the visibility assistance image generating device 1B of the present disclosure and the visibility assistance image MP displayed on the display device 13. As shown in FIG. Note that the visibility support image MP is an image corresponding to the output image I _out described above, and is displayed on the display device 13 . A driver or the like on board the own vehicle 100 sees the visibility support image MP displayed on the display device 13 . Since the appearance of the external world of the own vehicle 100 is included in the visibility support image MP, the visibility support image MP assists the visibility of the driver or the like.

As shown in FIG. 11(b), the visibility support image MP has a detected vehicle display portion D and an aspherical portion A. As shown in FIG. The detected vehicle display portion D is an area on the image in which the detected vehicle DC whose position is acquired by the approaching vehicle position calculation device 15 is displayed. On the other hand, the aspherical portion A is a region on the image corresponding to the aspherical portion M2 in the optical aspherical mirror described above.

Originally, the aspherical portion M2 in the optical aspherical mirror has a higher degree of compression than other portions of the optical mirror in order to widen the viewing angle reflected on the mirror. Similarly, the aspherical portion A of the visibility support image MP has a higher compression rate than the detected vehicle display portion D. However, on a CMS (camera monitoring system), the detected vehicle display section D does not necessarily have to be displayed at the same size. It is also possible to compress the detected vehicle display portion D at a constant compression rate and set the aspherical portion A at an even greater compression rate. In any case, in the visibility support image generation device 1B of the present disclosure, the total number of pixels to be compressed and the compression ratio around the detected vehicle DC are determined in advance.

(Regulations on labeling amount)
An example in which the compression ratio of the detected vehicle display portion D is set to 1 (same size) will be described below. First, the final display angle of view of the visibility support image MP that can be visually recognized by the driver or the like on the display device 13 is defined. In addition, as shown in FIG. 11A, the number of pixels on the sensor image may be specified.

In this example, the example is based on the number of pixels on the sensor image. The number of pixels in the horizontal direction on the sensor image shown in FIG. 11(a) is determined in advance to be 1300 pixels. The number of pixels in the horizontal direction of the detected vehicle display portion D on the sensor image is set to 500 pixels. The number of pixels in the horizontal direction of the aspherical portion A on the sensor image is defined as 800 pixels.

At this time, the area of 500+800 pixels in the horizontal direction shown in FIG. 11A on the sensor image is partially compressed and converted into the visibility support image MP. Assume that the number of pixels in the horizontal direction of the visibility support image MP is 800 pixels.

Note that the number of pixels described above is merely an example, and other set values may be used.

At 1300 pixels in the horizontal direction in the sensor image shown in FIG. 11A, the aspherical portion A is compressed from 800 pixels to 300 pixels. On the other hand, the 500 pixels of the detected vehicle display D are not compressed in this example. This is because the display magnification of the detected vehicle display portion D is determined to be 1:1 in advance. Then, as shown in FIG. 11(b), the view support image MP is generated with the aspherical portion A contracted in the horizontal direction. Note that the angle of view of 1300 pixels before compression is also maintained in the visibility support image MP.

Then, the proportional division ratio on the display (detected vehicle display portion D: aspherical portion A=500:300) as shown in FIG. 11(b) is fixed. The mode shown in FIG. 11B is assumed to be the initial display state. Next, the follow-up movement of the detected vehicle display portion D from this initial state will be described.

12A and 12B are diagrams showing the following in the view support image MP of the detected vehicle display section D, showing (a) initial state, (b) following state, and (c) following end state, respectively.

A dashed line shown on the right side of FIG. 12 on the view support image MP indicates a predetermined point that is a variable threshold. In this example, a variable threshold is virtually set at the center of the detected vehicle display section D in the lateral direction (horizontal direction) as shown in the figure. Note that the variable threshold may be shifted further to the right or left of the detected vehicle display section D.

The detected vehicle DC detected by the approaching vehicle position calculation device 15 is an overtaking vehicle that overtakes the host vehicle 100 from behind on the road. The detected vehicle DC is displayed so as to move from the left side to the right side on the detected vehicle display portion D shown in FIG.

After the detected vehicle DC (predetermined location, for example, the front surface of the vehicle (front bumper, etc.)) exceeds the above-described variable threshold, the detected vehicle display portion D moves to the right side in the visibility support image MP. The detected vehicle display portion D is moved so that the detected vehicle DC (predetermined portion thereof) is positioned at the center of the detected vehicle display portion D in the horizontal direction. That is, the position of the detected vehicle display portion D in the visibility support image MP moves rightward from the initial state (following state in the middle of FIG. 12) following (predetermined portion in) the detected vehicle DC.

Along with the following movement of the detected vehicle display portion D, the aspherical portion A is divided into two aspherical portions A1 and A2. However, the proportional division ratio on the display (detected vehicle display portion D: aspherical portion A=500:300) remains fixed. For example, the aspherical portion A1, which is the original aspherical portion A, present on the right side of the detected vehicle display portion D occupies an area of 150 pixels in the horizontal direction in the follow-up state shown in FIG. The additional aspherical portion A2 that appears to supplement the left side area of the detected vehicle display portion D occupies an area of 150 pixels in the horizontal direction in the following state shown in FIG. Together, the aspherical portions A1 and A2 are 300 pixels in the horizontal direction. In this manner, as the detected vehicle display portion D follows the movement, the number of pixels (angle of view) of the aspherical portions A1 and A2 arranged on the left and right sides of the detected vehicle display portion D also changes. The width of change in the number of pixels (angle of view) of the aspherical portions A1 and A2 is from 0 pixel (0 degree) to a predetermined design value.

The follow-up movement of the detected vehicle display portion D in the view support image MP as described above is performed until the follow-up end state shown in the lower part of FIG. 12 is reached. In the tracking end state, 300 pixels on the left side of the view support image MP are the aspherical portion A2, and 500 pixels on the right side are the detected vehicle display portion D. The width of the aspherical portion A1 is 0 pixels.

The detected vehicle DC overtakes the own vehicle 100 and exits in the right direction of the visibility support image MP. That is, the detected vehicle DC is no longer displayed in the visibility support image MP. When the detected vehicle DC is no longer displayed in the visibility support image MP, the display form of the visibility support image MP returns to the initial state shown in the upper part of FIG.

FIG. 13 is a comparison diagram between the conventional technology and the visibility support image generation device 1B of the present disclosure when the visibility support image is generated from the input image _Iin . FIG. 6(a) is the input image _Iin . FIG. 13B shows an output image I _out generated from _the input image I in using the conventional technique. A view support image MP (output image I _out ) generated using the view support image generation device 1B of the present disclosure for the same input image I _in is shown in FIG. 13(c).

13(a) and 13(b) correspond to FIGS. 8(a) and 8(b), respectively.

In FIG. 13B, since the right side portion corresponding to the aspherical portion of the output image I _out is compressed, not only the vehicle object OBJ1 but also the vehicle object OBJ2 are distorted in shape. Also, object OBJ2 appears to decelerate rapidly as it moves to the right side of the image, as described with reference to FIG.

On the other hand, in the case of the visibility support image MP (output image I _out ) generated using the visibility support image generation device 1B of the present disclosure, the shape of the detected vehicle DC (object OBJ2) is distorted. not This is because the detected vehicle DC is displayed in the detected vehicle display portion D. FIG. As described above, in this example, the detected vehicle display portion D is displayed at the same size. On the other hand, the left and right aspherical portions A1 and A2 are laterally compressed.

Therefore, in the visibility assistance image MP generated by the visibility assistance image generation device 1B, the detected vehicle DC, which has the highest possibility of collision and should be paid the most attention to, is displayed in the correct shape and at the correct position. In addition, the situation in which the detected vehicle DC approaches the own vehicle 100 from behind and overtakes the own vehicle 100 can be intuitively displayed at a wide angle of view by the visibility support image MP to present information to the occupants. can.

Next, allocation of the detected vehicle DC to the visibility support image MP will be described.

As described above, the view support image MP has the detected vehicle display portion D and the aspherical portion A1 (and A2). The aspherical portion A1 (and A2) is appropriately compressed to achieve a wide angle of view in the horizontal direction.

Here, there is a degree of freedom in laying out the detected vehicle DC on the view support image MP. FIG. 14 is a diagram showing one mode of allocation, showing (a) a sensor image and (b) a visibility support image MP.

In the sensor image shown in FIG. 14( a ), the entire 1300 pixels in the horizontal direction described above is the range in which the vehicles (detected vehicle DC and other vehicles) are displayed.

Here, two circles are plotted in FIG. 14(a). One indicates the position of the detected vehicle DC (object OBJ2), and the other indicates the position of the vehicle (object OBJ1) running in front of the detected vehicle DC. On the sensor image shown in FIG. 14(a), the aspherical portions A1 and A2 are still in a pre-compression state.

In the situation shown in FIG. 14A, from the left end of the sensor image to the detected vehicle DC (distance 1), from the detected vehicle DC to the object OBJ1 (distance 2), and from the object OBJ1 to the right end of the sensor image (distance 3). , are equally spaced.

Based on such a sensor image, the visibility support image generation device 1B shifts the display position of the detected vehicle DC onto the visibility support image MP so that the intervals 1, 2, and 3 remain equal. can be assigned. This assignment can be realized by appropriately adjusting the width of each of the aspherical portions A1 and A2 arranged on the left and right, and by appropriately adjusting the compression rate of each portion of the aspherical portions A1 and A2.

By assigning the display position of the detected vehicle DC as described above, the display position of the detected vehicle DC in the visibility support image MP is the maximum angle of view range (1300 pixels) that can be displayed in the visibility support image MP. Follow the way you move.

Next, even allocation based on polar coordinates will be described.

FIG. 15 is a diagram illustrating allocation based on polar coordinates. As shown on the right side of the figure, own vehicle 100 and detection vehicle DC are running side by side. The detected vehicle DC is traveling in a lane adjacent to the own vehicle 100 and is about to overtake the own vehicle 100 .

On the right side of FIG. 15, the field of view range (for example, the angle of view of 90 degrees) of the camera 12 provided on the vehicle 100 is shown in a fan shape. In addition, a plurality of interval lines PCL are drawn to divide the visual field range into equal intervals in the polar coordinate system.

On the other hand, on the left side of FIG. 15, the interval line PCL is drawn in the visibility support image MP (in this example, the horizontal angle of view is 90 degrees) displayed on the display device 13 .

Then, the display position of the detected vehicle DC in the visibility support image MP (left side of FIG. 15) may be assigned so as to match the relative position (right side of FIG. 15) between the camera 12 and the detected vehicle DC on the polar coordinates. This allocation can also be realized by appropriately adjusting the width of each of the aspherical portions A1 and A2 arranged on the left and right, and by appropriately adjusting the compression rate of each portion of the aspherical portions A1 and A2.

FIG. 16 is a processing flow diagram based on the system configuration shown in FIG.

In step S101, the adjacent vehicle position calculation device 15 confirms the vehicle information of the adjacent lane.

At step S102, the processing unit 11 determines whether or not there is a vehicle in the adjacent lane. If it exists (Yes), the process proceeds to step S103. If it does not exist (No), the visibility support image MP is displayed in the above-described initial state (see the upper part of FIG. 12) without following the detected vehicle display portion D. This display is performed by the display device 13 under the control of the processing section 11 .

In step S103, the processing unit 11 calculates the display position of this vehicle on the visibility support image MP based on the position information of the closest vehicle (detected vehicle DC) confirmed by the proximity vehicle position calculation device 15. FIG.

In step S104, the processing unit 11 determines whether or not the display position of the detected vehicle DC on the visibility assistance image MP has exceeded a predetermined location, which is a variable threshold value, on the visibility assistance image MP. If it exceeds (Yes), the process proceeds to step S105. If it does not exceed (No), the visibility support image MP is displayed in the above-described initial state (see the upper part of FIG. 12) without following the detected vehicle display portion D. FIG. This display is performed by the display device 13 under the control of the processing section 11 .

In step S105, the processing unit 11 forms display data (visual field support image MP) according to the display position of the detected vehicle DC on the visual field support image MP. That is, processing such as tracking of the detected vehicle display portion D as described above is performed.

With the above configuration, the visibility support image MP displayed on the display device 13 can be an image that does not give a sense of discomfort to the driver of the own vehicle 100 or the like, and also secures a wide angle of view due to the aspherical portion.

Next, for the purpose of the present disclosure, which is to provide a display image that gives less sense of incongruity and allows an appropriate grasp of the situation in the outside world, the visibility support image generation device 1C, which is a device for expressing a sense of distance on the display image, will be described. do.

Some vehicles have optical mirrors or replace the aforementioned optical aspherical mirrors with onboard cameras. The image captured by the in-vehicle camera is appropriately image-processed and displayed on the in-vehicle display device or the like as a visual field support image. A driver or the like in the vehicle can see the visual field support image to check the situation outside the vehicle.

Here, in the case of the optical mirror before replacement, the driver or the like can get a sense of distance from the movement of the eyeball caused by the parallax of the naked eye and the reaction of focusing on the object.

On the other hand, in-vehicle cameras generally do not have a mechanism for adjusting focus. Therefore, for example, when an image is captured at infinity with a fixed focus, the obtained image has a uniform focus. Therefore, the driver or the like cannot obtain a sufficient sense of perspective from the visual field support image displayed on the display device.

In particular, when the door mirror is replaced with an in-vehicle camera in a CMS (camera monitoring system), etc., the lack of perspective becomes a problem in scenes such as when the vehicle is backed up while looking at the image on the display device.

For example, consider the case where the driver reverses the vehicle to park. A parking frame may be partitioned off behind the vehicle by a fence or the like. At this time, from the driver's point of view, the image on the display device does not allow the driver or the like to obtain a sufficient sense of perspective, so it looks as if a fence is stuck in the vehicle body. In such a state, the driver cannot properly determine the stop position for reverse parking.

Therefore, in the visibility support image generation device 1C of the present disclosure, the gaze point of the person to the display device is grasped, the gaze point is focused, and the point is blurred at a different distance (perspective). Thus, a sense of distance can be virtually expressed on the view support image.

The configuration for virtually expressing the sense of distance on the visibility support image will be described below. With the visibility support image generation device 1C of the present disclosure, the driver or the like can obtain the sense of distance on the visibility support image. can.

FIG. 17 is a configuration diagram showing an example of a visibility support image generation device 1C of the present disclosure. Each device shown in FIG. 17 is mounted on the own vehicle 100 . The hardware configuration shown in FIG. 17 is basically the same as the visibility support image generation device 1 described based on FIG. The embodiment shown in FIG. 17 differs from the embodiment shown in FIG. 3 in that the view support image generation device 1C has two cameras and further includes a detection unit 18. be. The detection unit 18 will be described later, and first, the increase in the number of cameras to two will be described.

FIG. 18 shows various installation examples of an in-vehicle camera. 18, an orthogonal coordinate system is added in which the x-axis is the traveling direction of the vehicle, the y-axis is the width direction of the vehicle, and the z-axis is the height direction of the vehicle.

In the visibility support image generation device 1C of the present disclosure, two or more cameras are arranged in order to obtain a parallax detection range. In the example of FIG. 18, two cameras are arranged. The installation positions of multiple cameras are free. As shown in FIG. 18(a), one camera may be installed in each of the door mirrors and the rear of the vehicle. Also, two cameras may be installed at the rear of the vehicle. You may install two cameras in the door mirror of a vehicle.

Also, as shown in FIG. 18B, one camera may be installed in each of the door mirrors of the vehicle and the vicinity of the roof of the A pillar.

When two cameras are installed, one camera is the primary camera 12 and the other is the secondary camera 16 . An image captured by the main camera 12 is displayed on a display device 13 (monitor) of a CMS (camera monitoring system), and an image captured by the sub camera 16 is used for parallax detection. However, the image of the secondary camera 16 may be displayed on the display device 13 (monitor) of the CMS (camera monitoring system).

Then, the distance to the subject is estimated using the image of the sub-camera 16 with respect to the pixels in the image captured by the main camera 12 where the distance can be estimated. This distance estimation may use existing distance estimation techniques.

Since it is not always necessary to estimate the distance for all the pixels in the image captured by the main camera 12, the pixels on the main camera 12 may be thinned out to estimate the distance. This saves the time required for image processing.

Next, the detection unit 18 will be described with reference to FIG. 17 again. The detection unit 18 is typically a camera for monitoring the eyeballs of a driver or the like provided near the display device 13 of a CMS (camera monitoring system) or near the display device 13 . For example, since the driver of the vehicle looks at the display device 13 while driving, it is possible to monitor the eyeballs with a camera and detect where the driver is gazing on the display of the display device (gazing point).

The processing unit 11 that has obtained information related to the gaze point from the detection unit 18 estimates gaze coordinates (coordinates corresponding to the gaze point) in the visibility support image MP displayed on the display device 13 .

Then, the processing unit 11 applies a blurring filter according to the distance (perspective) calculated by the above-described distance estimation performed by the main camera 12 and the sub camera 16, centered on the gaze coordinates in the view support image MP, to the field of view. Apply to the support image MP. Transmission superimposition of white shading may be performed.

More specifically, the blurring filter is applied to pixels whose distance from the gaze coordinates exceeds a predetermined threshold (pixels whose perspective is significantly different from the gaze coordinates). Also, the larger the distance, the stronger the blurring filter may be applied.

With the above configuration, a fixed area (area within a threshold distance) that is close to the fixation point of the visibility support image is in focus, and the surrounding area becomes blurred as the estimated distance increases from the fixation point. be in a strong state. Therefore, when the driver or the like views the visibility support image MP using the display device 13, it is possible to obtain a sense of distance between the gaze target and the others.

Note that the predetermined threshold value (for example, 10 meters) and the distance resolution for changing the blur intensity can be changed according to the accuracy of distance detection based on the parallax of a plurality of cameras and the specifications. Also, the blurring filter may be applied in units of blocks composed of a plurality of pixels instead of in units of pixels.

As a modification, the touch panel display device 13 may be used to change/fix the fixation point by touching the touch panel screen with a finger or the like.

FIG. 19 is a processing flow diagram based on the system configuration shown in FIG. 17, showing (a) processing for representing a virtual sense of distance, and (b) brightness control processing for the display device 13 .

In step S201, the processing unit 11 calculates the distance information of the main camera 12 based on the images captured by the main camera 12 and the sub camera 16 respectively.

In step S202, the detection unit 18 checks whether the driver or the like is gazing at the display device 13 (monitor). If the user is watching (Yes), the process proceeds to step S203. If the user is not paying attention (if No), the process ends.

In step S203, the processing unit 11 calculates the gaze coordinates described above. An image captured by the main camera 12 and information related to the gaze point detected by the detection unit 18 are used for this calculation processing.

In step S204, the processing unit 11 applies a blur filter corresponding to the distance information in the gaze coordinates and the surrounding distance information (calculated in step S201) to the visibility support image MP. In the visibility support image MP after application of this filter, a sense of distance is expressed in a pseudo manner.

Next, the luminance dimming process shown in FIG. 19(b) will be described. As described above, the detection unit 18 shown in FIG. 17 monitors the eyeballs of the driver or the like with a camera, and calculates where the driver is gazing on the display of the display device 13 (monitor). Therefore, the display device 13 (monitor) is required to have a brightness that enables appropriate eyeball monitoring.

In step S301, the detection unit 18 checks whether the driver or the like is gazing at the display device 13 (monitor). If the user is watching (Yes), the process proceeds to step S302. If not gazed (No), the process proceeds to step S304.

In step S302, the processing unit 11 determines whether the display device 13 (monitor) has reached the target luminance. If the target luminance has been reached (Yes), there is no need to further increase the luminance, and the process ends. If the target luminance is not reached (No), the process proceeds to step S303 to gradually increase the luminance of the display device 13 (monitor).

Step S304 is a process performed when the driver or the like is not looking at the display device 13 (monitor) (see step S301). At this time, the detection unit 18 determines whether or not the non-gazing state continues for a certain period of time. If the non-gazing state continues for a certain period of time (Yes), the process proceeds to step S305 to dim the display device 13 (monitor). On the other hand, if the non-gazing state has not continued for a certain period of time (if No), the light reduction process is not performed, and the process ends. This is because when the driver takes his or her eyes off the display device 13 (monitor) many times in a short period of time, if the light is dimmed each time, the brightness will repeatedly fluctuate in a short period of time, which is annoying.

FIG. 20 is a comparison diagram of the visibility support image displayed on the display device 13, (a) a diagram showing the gaze point on the display device, (b) when the visibility support image generation device 1C of the present disclosure is not used Visibility support image, (c) is a visibility support image MP when the visibility support image generation device 1C of the present disclosure is used.

In this example, the driver's gaze point is in the white circle portion shown in FIG. 20(a). Based on this, when comparing FIG. 20(b) and FIG. 20(c), in FIG. 20(b), since the entire image is in focus, it is difficult to sense the distance. On the other hand, in FIG. 20(c), a pseudo sense of distance is expressed by blurring a portion at a certain distance or more from the fixation point in the visibility support image. As a result, it is possible to give a sense of security to the driver or the like who sees the view support image.

In addition, although illustration is omitted, a plurality of buildings (three-dimensional objects) may appear in the view support image. For example, a distant building and a nearby building are reflected. Here, a building at a distance similar to the gaze coordinates is in focus even after the blurring filter is applied. In other words, the driver or the like can recognize which building is at the distance of the degree of identification. This point can also be said to be one of the effects of the visibility support image generation device 1C of the present disclosure.

In the above configuration, in the image conversion, the compression rate changes linearly as the declination angle changes from the horizontal direction of the captured image to the vertical direction of the captured image, centering on a deep vanishing point included in the captured image. It may be something to do. In the case of linear change, the compression ratio gradually changes according to the deflection angle, so the generated view support image also becomes natural.

In the above configuration, the image conversion may increase the compression rate as the distance from the deep vanishing point increases. With this configuration, it is possible to naturally widen the angle of view while maintaining the information amount of the image by low-compression of the peripheral portion of the deep vanishing point.

In the above configuration, the image conversion may be compression using an elongated elliptical lens model. The vertically elongated elliptical model has a shape similar to that of a perfectly circular lens model that is normally used, and there is little discomfort in the displayed image after image conversion. Also, while the perfect circle is defined by only one parameter, the radius r, a vertical ellipse can be defined by two parameters, the major axis b and the minor axis c. By appropriately adjusting these two parameters, it is possible to flexibly change the vertical and horizontal compression ratios of the image while reducing the discomfort of the displayed image that has conventionally occurred.

In the above configuration, the visibility assistance image generation device 1 may further include a display unit that displays the visibility assistance image. By displaying the visual field support image on such a display unit, a driver or the like can view the natural visual field support image with a wider field of view in the horizontal direction.

The present disclosure also relates to an image conversion program for generating a visibility assistance image of a vehicle. The image conversion program instructs the processing unit of the apparatus to convert the real image height of the image and the geometric lens model based on the data indicating the correspondence relationship between the real image height of the image and the ideal image height of the perfect circular lens model. calculating the correspondence relationship between the ideal image height and the compression ratio of each pixel included in the image based on the correspondence relationship between the real image height of the image and the ideal image height of the geometric lens model and compressing the image based on the compression rate of each pixel, wherein the geometric shape has a length laterally from the center of the geometric shape that is equal to the geometric shape It may be shorter than the length in the longitudinal direction from the center of the shape. According to the above configuration, it is possible to provide a visual field support image from an input image captured by a camera provided in a door mirror or the like of a vehicle so that the driver or the like is less uncomfortable and can appropriately grasp the situation of the outside world.

In the above configuration, the geometric shape may vary linearly as the distance from the center of the geometric shape changes as the deflection angle changes from the horizontal direction to the vertical direction of the geometric shape. If it is a linear change, the compression rate will gradually change according to the angle of argument, so the image generated using the program will also look natural to the driver.

In the above configuration, the geometric shape may be an elongated ellipse. The vertically elongated elliptical model has a shape similar to that of a perfectly circular lens model that is normally used, and there is less sense of discomfort in the field-of-view support image after image conversion. Also, while the perfect circle is defined by only one parameter, the radius r, a vertical ellipse can be defined by two parameters, the major axis b and the minor axis c. By appropriately adjusting these two parameters, it is possible to flexibly change the vertical and horizontal compression ratios of the image while reducing the discomfort of the displayed image that has conventionally occurred.

In addition, the vehicle includes a visibility support image generation device, a display device, and an approaching vehicle position calculation device, and the approaching vehicle position calculation device is the closest vehicle traveling in a lane adjacent to the lane in which the vehicle, which is the own vehicle, travels. Acquiring position information of a detected vehicle that is a nearby vehicle, the visibility support image generation device includes a camera that captures an image from the own vehicle, and a processing unit, and the processing unit captures the image captured by the camera. An image is image-converted to generate a visibility assistance image, the display device displays the visibility assistance image, the visibility assistance image has a detected vehicle display portion and an aspherical portion, and the aspherical portion The compression rate is higher than the compression rate in the detected vehicle display section, the detected vehicle is displayed in the detected vehicle display section, and the display position of the detected vehicle in the detected vehicle display section is the same as that in the detected vehicle display section. After exceeding the predetermined position, the position of the detected vehicle display portion in the visibility support image may follow the movement of the display position of the detected vehicle.

Similarly, a visibility support image generation device for generating a visibility support image of a vehicle includes a camera that captures an image from the vehicle and a processing unit, and the processing unit is the own vehicle from the nearby vehicle position calculation device. Acquires the position information of the detected vehicle, which is the closest vehicle traveling in a lane adjacent to the lane in which the vehicle travels, and the processing unit converts the captured image captured by the camera into an image to support visual field for display. generating an image, wherein the visibility assistance image has a detected vehicle display portion and an aspherical portion, wherein a compression ratio in the aspherical portion is higher than a compression ratio in the detected vehicle display portion, and the detected vehicle is the After the display position of the detected vehicle in the detected vehicle display portion exceeds a predetermined position in the detected vehicle display portion, the position of the detected vehicle display portion in the view support image is displayed on the detected vehicle display portion. may follow the movement of the display position of the detected vehicle.

With the above configuration, it is possible to provide a visual field support image that does not cause discomfort to the driver or the like. Furthermore, since the visibility support image still has a portion corresponding to the aspherical portion, it is possible to display with a wide angle of view.

In the above configuration, the visibility assistance image may have two aspherical portions, and the two aspherical portions may be arranged on the left and right sides of the detected vehicle display portion in the visibility assistance image. Further, as the position of the detected vehicle display portion in the view support image follows the movement of the display position of the detected vehicle, the angle of view or the number of pixels in the two aspherical portions changes, and the two The angle of view or the number of pixels in the aspherical portion may be changed so that the angle of view or the number of pixels in the visibility support image is maintained at a predetermined value. With the above configuration, even if the position of the detected vehicle display portion follows the movement of the display position of the detected vehicle, the second aspherical portion can compensate for the angle of view of the visibility support image.

In the above configuration, the display position of the detected vehicle in the visibility support image may be assigned so as to follow the movement of the detected vehicle in the maximum view angle range displayable in the visibility support image. With the above configuration, it is possible to display the movement of the detected vehicle in the visibility support image without causing a sense of incongruity.

In the above configuration, the display position of the detected vehicle in the visibility support image may be assigned so as to match the relative position between the camera and the detected vehicle on polar coordinates. With the above configuration, the angular position of the detected vehicle in the visual field support image is displayed at the same position as seen by the occupant of the vehicle with the naked eye, so there is little sense of discomfort.

Further, in a vehicle equipped with a visibility support image generation device, the visibility support image generation device includes a processing unit, a first camera, a second camera, and a visibility support image generated based on an image captured by the first camera. a display device that displays an image; and a detection unit that detects a gaze point of an occupant on the vehicle in the display device, and the processing unit detects the subject from the first camera in the image captured by the first camera. The distance to is calculated using the parallax generated between the first camera and the second camera, and the detection unit detects the gaze point on the display device of the occupant riding in the vehicle, and the processing unit estimates the occupant's gaze coordinates in the visibility support image based on the gaze point, and the processing unit applies a blur filter based on the distance centered on the gaze coordinates in the visibility assistance image , may be applied to the visibility assistance image.

Similarly, a visibility support image generation device that generates a visibility support image of the vehicle displays the visibility support image generated based on the images captured by the processing unit, the first camera, the second camera, and the first camera. and a detection unit that detects a gaze point of an occupant in the vehicle on the display device, and the processing unit detects the distance from the first camera to the subject in the image captured by the first camera. is calculated using the parallax generated between the first camera and the second camera, the detection unit detects a gaze point on the display device of an occupant riding in the vehicle, and the processing unit calculates the Based on the gaze point, the occupant's gaze coordinates in the visibility support image are estimated, and the processing unit applies a blur filter according to the distance centered on the gaze coordinates in the visibility support image to the field of view. May be applied to support images.

With the above configuration, a driver or the like in a vehicle can obtain a sense of distance on the visibility support image.

In the above configuration, the blurring filter may apply blurring to pixels or pixel blocks at a predetermined distance or more with respect to the gaze coordinates. Further, the blurring filter may apply stronger blurring as the distance from the gaze coordinate increases. With the above-described configuration, a point close to the gaze coordinates is in focus, and a point distant from the gaze coordinates is out of focus. can be obtained more clearly. Further, by applying stronger blurring as the distance from the gaze coordinate increases, the driver's attention is focused on the vicinity of the gaze coordinate, and the driver can obtain a clearer sense of distance.

In the above configuration, the resolution of the distance of the blurring filter may be set according to the parallax detection accuracy of the first camera and the second camera. With the above configuration, when the parallax detection accuracy of a plurality of cameras is high, it is possible to express a more detailed sense of distance on the view support image.

In the above configuration, the luminance of the display device may be reduced when the detection unit detects that the occupant's gaze point is not on the display device. Further, when the detection unit detects that the occupant's gaze point is on the display device, the processing unit may gradually increase the brightness of the display device. With the above configuration, it is possible to appropriately detect the gaze point while appropriately controlling the brightness of the display device depending on whether or not the vehicle occupant is looking at the display device.

Various embodiments have been described above with reference to the drawings, but it goes without saying that the present invention is not limited to such examples. It is obvious that a person skilled in the art can conceive of various modifications or modifications within the scope described in the claims, and these also belong to the technical scope of the present invention. Understood. Moreover, each component in the above embodiments may be combined arbitrarily without departing from the spirit of the invention.

1 visibility assistance image generation device 1B visibility assistance image generation device 1C visibility assistance image generation device 11 processing unit 12 camera 13 display device 14 nonvolatile memory 15 proximity vehicle position calculation device 16 camera 18 detection unit 50 lens 51 evaluation surface 52 real image height 53 Ideal image height 100 Own vehicle A Aspherical part A1 Aspherical part A2 Aspherical part D Detected vehicle display part E Vertical elliptical lens model I _In input image I _out output image M Door mirror M1 Mirror part M2 Aspherical part MP Visibility support images OBJ1 to OBJ4 Object PCL interval line

Claims (12)

A vehicle equipped with a vision support image generation device,
The visibility support image generation device,
a processing unit;
a first camera, a second camera,
a display device that displays a visibility support image generated based on an image captured by the first camera;
a detection unit that detects a gaze point on the display device of an occupant riding in the vehicle;
with
The processing unit calculates the distance from the first camera to the subject in the image captured by the first camera using the parallax generated between the first camera and the second camera,
The detection unit detects a gaze point on the display device of an occupant riding in the vehicle,
The processing unit estimates gaze coordinates of the occupant in the visibility support image based on the gaze point,
The processing unit applies a blurring filter according to the distance centered on the gaze coordinate in the view support image to the view support image.
vehicle. A vehicle according to claim 1,
The blurring filter applies blurring to pixels or pixel blocks whose distance is equal to or greater than a predetermined threshold with respect to the gaze coordinates.
vehicle. A vehicle according to claim 1 or claim 2,
The blurring filter applies stronger blurring as the distance based on the gaze coordinates increases.
vehicle. A vehicle according to any one of claims 1 to 3,
The resolution of the distance of the blurring filter is set according to the parallax detection accuracy of the first camera and the second camera.
vehicle. A vehicle according to any one of claims 1 to 4,
When the detection unit detects that the occupant's gaze point is not on the display device, the brightness of the display device is reduced.
vehicle. A vehicle according to any one of claims 1 to 5,
When the detection unit detects that the gaze point of the occupant is on the display device, the processing unit gradually increases the brightness of the display device.
vehicle. A visibility support image generation device for generating a visibility support image of a vehicle,
a processing unit;
a first camera, a second camera,
a display device that displays a visibility support image generated based on an image captured by the first camera;
a detection unit that detects a gaze point on the display device of an occupant riding in the vehicle;
with
The processing unit calculates the distance from the first camera to the subject in the image captured by the first camera using the parallax generated between the first camera and the second camera,
The detection unit detects a gaze point on the display device of an occupant riding in the vehicle,
The processing unit estimates gaze coordinates of the occupant in the visibility support image based on the gaze point,
The processing unit applies a blurring filter according to the distance centered on the gaze coordinate in the view support image to the view support image.
Vision support image generation device. The visibility support image generation device according to claim 7,
The blurring filter applies blurring to pixels or pixel blocks whose distance is equal to or greater than a predetermined threshold with respect to the gaze coordinates.
Vision support image generation device. The visibility support image generation device according to claim 7 or claim 8,
The blurring filter applies stronger blurring as the distance increases based on the gaze coordinates.
Vision support image generation device. The visibility support image generation device according to any one of claims 7 to 9,
The resolution of the distance of the blurring filter is set according to the parallax detection accuracy of the first camera and the second camera.
Vision support image generation device. The visibility support image generation device according to any one of claims 7 to 10,
When the detection unit detects that the occupant's gaze point is not on the display device, the brightness of the display device is reduced.
Vision support image generation device. The visibility support image generation device according to any one of claims 7 to 11,
When the detection unit detects that the gaze point of the occupant is on the display device, the processing unit gradually increases the brightness of the display device.
Vision support image generation device.

Images