JP6585543B2 - Driver visual information providing device and vehicle driving support system - Google Patents

Description

  The present invention relates to a driver visual information providing apparatus that outputs information related to the visual state of a driver who is driving an automobile, and a vehicle driving support system that notifies the driver of the presence of an obstacle ahead of the host vehicle based on the information. .

  Conventionally, there has been known a vehicle driving support system that alerts a driver to notify the presence of an obstacle when an image of the front of the host vehicle is captured by a camera and an obstacle is detected based on the captured image. It has been. However, the driver feels bothered by such alerting when he / she recognizes the obstacle. Therefore, in the apparatus proposed in Patent Document 1, an obstacle is detected within a predetermined range deviated from the direction of the driver's line of sight, and the direction of the driver's line of sight matches the steering angle direction of the steering wheel. Limited attention is given.

JP 2011-113120 A

  In such a vehicle driving support system, the visual ability of each driver is not considered. For example, when a plurality of drivers see a common landscape, each driver has a different ability to see the landscape. In addition, there are areas with relatively high visual ability and areas with low visual ability within the visual field of the driver, and the distribution of such areas varies depending on the driver. For this reason, even in the same situation, an obstacle that can be recognized well for a certain driver may be overlooked by another driver.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to enable appropriate alerting according to the visual ability of each driver.

In order to achieve the above object, the features of the present invention are:
Photographing means (21) for photographing the outside world ahead of the host vehicle;
Line-of-sight detection means (11, 22) for detecting the direction of the line of sight of the driver of the host vehicle;
Acquire individual eye characteristics that represent at least one of the characteristics of the eyes of the driver, including eyeglasses or contact lenses, and the characteristics of the retina of the driver. Personal eye characteristic acquisition means (15, 23),
Based on the external image obtained by photographing by the photographing means, the direction of the visual line of the driver detected by the visual line detecting means, and the personal eye characteristics acquired by the personal eye characteristic acquiring means, the visual field of the driver Map calculation means (12, 13, 14) for calculating an oversight risk map that associates a position in the vehicle with an oversight risk that is an index of ease of oversight of the driver, and calculated by the map calculation means It is to output information representing an oversight risk map.

  The driver visual information providing apparatus of the present invention includes a photographing unit, a line-of-sight detection unit, a personal eye characteristic acquisition unit, and a map calculation unit. The photographing means photographs the outside world ahead of the host vehicle. The line-of-sight detection means detects the direction of the driver's line of sight. The direction of the driver's line of sight can be determined by detecting, for example, the driver's head movement, eyeball movement, and the like.

The personal eye characteristic acquisition means is an eye characteristic that affects a driver's ability to view the outside world in each driver , and is at least one of a driver's eye optical system characteristics including glasses or contact lenses, and a driver's retinal characteristics. Acquire personal eye characteristics representing. As the personal eye characteristic acquisition means, for example, a storage device that inputs and stores the personal eye characteristic of the driver can be used.

  The map calculation means is based on the external image obtained by the photographing by the photographing means, the direction of the driver's line of sight detected by the line-of-sight detection means, and the personal eye characteristic acquired by the personal eye characteristic acquisition means. An oversight risk map is calculated by associating the position in the field of view with the oversight risk that is an index of the driver's ease of oversight. Therefore, the easy-to-miss direction in the driver's field of view can be ascertained from the oversight risk map. The oversight risk is information that represents an index representing ease of oversight in two or more stages (multiple stages).

  The driver visual information providing device outputs information representing an oversight risk degree map calculated by the map calculation means. Therefore, by acquiring the information output from the driver visual information providing device, it is possible to sequentially grasp the external world visual situation representing how the driver is viewing the external world. For example, by inputting information representing this oversight risk map to a vehicle driving support device or the like mounted on a vehicle, it is possible to alert an obstacle according to the external visual situation of each driver.

  This driver visual information providing device may be incorporated in an in-vehicle device (for example, a vehicle driving support device), or provided separately from the in-vehicle device, and represents an oversight risk map for the in-vehicle device. It may be provided to transmit information.

A feature of one aspect of the present invention is that
The personal eye characteristic acquisition means is configured to acquire an eye optical system characteristic of a driver including glasses or a contact lens,
The map calculation means includes:
Based on the external image obtained by photographing by the photographing means, the direction of the driver's line of sight detected by the line of sight detecting means, and the driver's eye optical system characteristics acquired by the personal eye characteristic acquiring means, the driver Retinal projection image estimation means (12) for estimating a retinal projection image which is an image of the external world projected onto the retina of
When a retinal projection image estimated by the retinal projection image estimation means is projected onto the retinal photoreceptor array model using a retinal photoreceptor array model that is a model in which retinal photoreceptors are arrayed, the retinal photoreceptor cell Retinal output image estimation means (13) for estimating a retinal output image which is an image obtained by the reaction of the array model;
By calculating the amount of information for each local area of a preset area in the retinal output image over the entire retinal output image, the position in the driver's field of view and the oversight that increases as the amount of information decreases And a local area information amount calculation means (14) for calculating an oversight risk level map that associates the risk level.

In one aspect of the present invention, the retinal projection image estimation means is a retinal projection image that is an image of the outside world projected on the driver's retina based on the outside world image, the driver's line-of-sight direction, and the driver's eye optical system characteristics. Is estimated. Accordingly, this retinal projection image reflects the characteristics of the eye optical system of each driver. When the retinal projection image estimated by the retinal projection image estimation means is projected onto the retinal photoreceptor array model, the retinal model output image estimation means uses the retinal photoreceptor array model, which is a model in which retinal photoreceptors are arrayed. Next, a retinal output image, which is an image obtained by the reaction of the retinal photoreceptor cell array model, is estimated.

  As this retinal photoreceptor cell arrangement model, for example, a retina in which photoreceptor cells (cone cells (red cones, green cones, blue cones) and rod cells) constituting the retina are arranged in the same manner as the human retina. Can be used. Therefore, when a retinal projection image is projected onto the retinal photoreceptor array model, a response (signal) corresponding to the retinal projection image of the retinal photoreceptor array model can be estimated. As a result, a retinal output image corresponding to the personal eye characteristics of the driver is obtained.

  The local area information amount calculating means calculates an information amount for each local area having a preset width in the retinal output image over the entire retinal output image. This amount of information is obtained, for example, by a function using the ratio of pixel values included in an image in each local region, and becomes larger as an image in which the normalized gradation distribution of pixel values is expanded. The local area information amount calculation means calculates an oversight risk map that associates the position in the driver's field of view with the oversight risk level that increases as the information amount decreases, based on the calculated information amount in each local area. To do. As a result, according to one aspect of the present invention, it is possible to calculate an oversight risk degree map that appropriately represents the visual state of each driver.

  In addition, according to one aspect of the present invention, the oversight risk map is calculated according to the characteristics of the eye optical system of the driver including glasses or contact lenses, so that the driver is wearing glasses or contact lenses. However, it is possible to calculate an oversight risk degree map that appropriately represents the visual state of the driver in that state.

A feature of one aspect of the present invention is that
The personal eye characteristic acquisition means is configured to acquire a retina characteristic of a driver;
The map calculation means includes:
A retina that estimates a retinal projection image that is an image of the outside world projected on the retina of the driver, based on the external image obtained by the photographing by the photographing unit and the direction of the driver's visual line detected by the visual line detecting unit. Projection image estimation means (12);
A model in which retinal photoreceptor cells are arranged, and estimated by the retinal projection image estimation means using a retinal photoreceptor array model set according to the retinal characteristics of the driver acquired by the personal eye characteristics acquisition means A retinal output image estimation means (13) for estimating a retinal output image that is an image obtained by a reaction of the retinal photoreceptor array model when a retinal projection image is projected onto the retinal photoreceptor array model;
By calculating the amount of information for each local area of a preset area in the retinal output image over the entire retinal output image, the position in the driver's field of view and the oversight that increases as the amount of information decreases And a local area information amount calculation means (14) for calculating an oversight risk level map that associates the risk level.

  According to one aspect of the present invention, the retinal photoreceptor cell arrangement model is set according to the retinal characteristics of each driver. Thereby, for example, a retinal characteristic representing a visual field defect region or the like can be overlooked and reflected in the risk map. Therefore, according to one aspect of the present invention, it is possible to calculate an oversight risk degree map that appropriately represents the visual state of each driver even for a driver with retinal disease.

A feature of one aspect of the present invention is that
The personal eye characteristic acquisition means is configured to acquire an eye optical system characteristic of a driver including glasses or a contact lens, and a retina characteristic of the driver,
The map calculation means includes:
Based on the external image obtained by photographing by the photographing means, the direction of the driver's line of sight detected by the line of sight detecting means, and the driver's eye optical system characteristics acquired by the personal eye characteristic acquiring means, the driver Retinal projection image estimation means (12) for estimating a retinal projection image which is an image of the external world projected onto the retina of
A model in which retinal photoreceptor cells are arranged, and estimated by the retinal projection image estimation means using a retinal photoreceptor array model set according to the retinal characteristics of the driver acquired by the personal eye characteristics acquisition means A retinal output image estimation means (13) for estimating a retinal output image that is an image obtained by a reaction of the retinal photoreceptor array model when a retinal projection image is projected onto the retinal photoreceptor array model;
By calculating the amount of information for each local area of a preset area in the retinal output image over the entire retinal output image, the position in the driver's field of view and the oversight that increases as the amount of information decreases And a local area information amount calculation means (14) for calculating an oversight risk level map that associates the risk level.

  According to one aspect of the present invention, the driver's eye optical system characteristics are reflected in the estimation of the retinal projection image, and the driver's retinal characteristics are reflected in the estimation of the retinal output image. As a result, the driver's eye optical system characteristics and retinal characteristics can be reflected in the oversight risk map. Therefore, according to one aspect of the present invention, it is possible to calculate an oversight risk degree map that more appropriately represents the visual state of each driver.

A feature of one aspect of the present invention is that
Pupil detection means (11) for detecting the size of the driver's pupil;
The retinal projection image estimation means (12) is configured to estimate the retinal projection image in consideration of the size of the pupil.

  According to one aspect of the present invention, since the retinal projection image is calculated in consideration of the size of the pupil (pupil diameter), a more appropriate oversight risk map can be calculated.

  The present invention can be applied not only to the driver visual information providing apparatus but also to a vehicle driving support system including the driver visual information providing apparatus.

In this case, the feature of the vehicle driving support system of the present invention is:
Obstacle information acquisition means (16) for acquiring obstacle information relating to the position of an obstacle existing in front of the host vehicle;
Based on the obstacle information acquired by the obstacle information acquisition means and the information representing the oversight risk map output from the driver visual information providing device, the driver is notified of the presence of the obstacle. And means (17).

  In the present invention, the obstacle information acquisition means acquires the obstacle information related to the position of the obstacle existing in front of the host vehicle. In this case, for example, the obstacle information acquisition unit may recognize the obstacle based on an external image obtained by shooting by the shooting unit and acquire relative position information of the obstacle with respect to the host vehicle.

  The alerting means notifies the driver of the presence of the obstacle based on the obstacle information and the information indicating the oversight risk map. Therefore, according to the present invention, it is possible to perform appropriate alerting according to the visual ability of each driver. That is, it is possible to call attention to an obstacle that is estimated to be overlooked by the driver.

A feature of one aspect of the present invention is that
The alerting means is configured to notify the driver of the presence of only an obstacle in the visual field region where the risk of oversight is higher than a threshold (S12: Yes).

  According to one aspect of the present invention, the driver is notified of the presence of only an obstacle in the visual field region whose risk of oversight is higher than the threshold. Therefore, it is possible to perform appropriate alerting according to the visual ability of each driver, and to reduce unnecessary alerting for obstacles recognized by the driver.

A feature of one aspect of the present invention is that
The alerting means is configured to notify the driver of the presence of an obstacle when a preset alerting condition is satisfied, and for an obstacle in the visual field region where the risk of oversight is higher than a threshold value Compared with the alerting condition (S142) for the obstacle in the visual field area where the oversight risk is equal to or less than the threshold, the alerting condition (S141) is easier to notify the driver of the presence of the obstacle. It is set to be.

  According to one aspect of the present invention, the presence of an obstacle is notified to the driver when a preset alerting condition is satisfied. This alerting condition is different between an obstacle in the visual field area where the risk of oversight is higher than the threshold and an obstacle in the visual field area where the risk of oversight is less than the threshold. Specifically, the alert conditions for obstacles in the visual field where the risk of oversight is higher than the threshold is higher than the alert conditions for obstacles in the visual field where the risk of oversight is less than the threshold. It is set so that it is easy to notify the driver of the presence of an obstacle. For this reason, it is easy to call attention to obstacles that are easily overlooked by the driver, and it is difficult to call attention to obstacles that are difficult for the driver to overlook. Accordingly, it is possible to perform appropriate alerting according to the visual ability of each driver, and to reduce unnecessary alerts regarding obstacles recognized by the driver.

  The alert condition is, for example, that the predicted collision time, which is the time from the current time to the time when the vehicle and the obstacle are predicted to collide, is shorter than the set time, or there is a possibility of collision with the vehicle It can be arbitrarily set such that the distance between the obstacle and the host vehicle is shorter than the set distance. In this case, for example, when the predicted collision time is used as a warning condition, the set time for an obstacle in the visual field area where the risk of oversight is higher than the threshold is set in the visual field area where the risk of oversight is less than the threshold. What is necessary is just to set longer than the set time with respect to the obstruction which is. When using the distance between an obstacle and the vehicle as a warning condition, set the distance to the obstacle in the visual field where the risk of oversight is higher than the threshold. What is necessary is just to make it longer than the setting distance with respect to the obstacle which has entered. As a result, it is easy to call attention to obstacles that are easily overlooked by the driver (attention is given early), and it is difficult to call attention to obstacles that are difficult for the driver to overlook.

A feature of one aspect of the present invention is that
The alerting means (S151) for notifying the driver of the presence of an obstacle in the visual field area where the risk of oversight is higher than a threshold is provided in the visual field area where the risk of oversight is below the threshold. Compared to the alerting mode (S152) for notifying the driver of the presence of an obstacle that is present, the alerting power to the driver is increased.

  According to one aspect of the present invention, when notifying the driver of the presence of an obstacle in the visual field where the risk of oversight is higher than the threshold, the obstacle in the visual field of which the risk of oversight is less than or equal to the threshold Compared to notifying the driver of the presence of an object, the driver's alerting power is increased. For this reason, strong attention is given to obstacles that are easily overlooked by the driver. On the other hand, for an obstacle that is difficult for the driver to overlook, a light alert is given or no alert is given. For this reason, even if the driver is alerted when the driver recognizes the obstacle, the driver feels less troublesome. Accordingly, it is possible to perform appropriate alerting according to the visual ability of each driver.

  In the above description, in order to assist the understanding of the invention, the reference numerals used in the embodiments are attached to the constituent elements of the invention corresponding to the embodiments in parentheses. It is not limited to the embodiment defined by.

1 is a schematic configuration diagram of a vehicle driving support system according to an embodiment. It is a figure showing an example of an eyeball optical system characteristic (wavefront aberration). It is a figure showing an example of a retina projection image. It is explanatory drawing explaining a retinal photoreceptor cell arrangement | sequence model. It is a graph showing the response dynamics of a photoreceptor cell. It is a figure showing an external field image and a retina output image. It is a graph showing an example of an external field image local quantity area normalization gradation distribution. It is explanatory drawing explaining the sampling of the information content in a local area | region. It is a figure showing an oversight risk degree map. It is a figure showing an example of the oversight risk degree map (one eye) represented using the light and shade. It is a figure showing an example of the oversight risk degree map (one eye) represented using the light and shade. It is a figure showing an example which piled up the oversight risk degree map (one eye) and the external image which were expressed using the light and shade. It is a figure showing an example of the oversight risk degree map (binocular) represented using the light and shade. It is a flowchart showing the alerting control routine which concerns on embodiment. 10 is a flowchart showing an attention control routine according to Modification Example 1. 10 is a flowchart illustrating a reminder control routine according to Modification 2.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a schematic configuration diagram of a vehicle driving support system 1 of the present embodiment.

  The vehicle driving support system 1 includes a support ECU 10. ECU is an abbreviation for Electric Control Unit. In this specification, when it is necessary to distinguish a vehicle on which this vehicle driving support system is mounted from other vehicles, it is referred to as “own vehicle”.

  The support ECU 10 is connected to the first camera 21, the second camera 22, the input unit 23, and the alarm device 24. The first camera 21 is fixed to the vehicle body, images the outside world ahead of the host vehicle at a predetermined short period, and outputs an outside world image obtained by the imaging to the support ECU 10. The first camera 21 corresponds to the photographing means of the present invention. The second camera 22 is mounted and fixed on the head of the driver, images the driver's face at a predetermined short cycle, and outputs the driver's face image obtained by the imaging to the support ECU 10. Further, the second camera 22 incorporates a gyro / acceleration sensor (not shown), and outputs information representing the movement state of the driver's head detected by the gyro / acceleration sensor to the support ECU 10. The input unit 23 is a device that inputs data representing the unique eye characteristics (referred to as individual eye characteristics) of the driver driving the host vehicle. The input unit 23 outputs the input personal eye characteristic to the support ECU 10.

  The alarm device 24 includes a display 24a and a speaker 24b (may be a buzzer), and is a device that is driven by the support ECU 10 and alerts the driver. A head-up display (HUD) is used as the display 24a. Hereinafter, the display 24a is referred to as HUD 24a. The HUD 24a receives display information from various ECUs and navigation devices in the host vehicle, and displays the display information on a partial area (display area) of the windshield of the host vehicle. The assistance ECU 10 transmits display information for calling attention to the HUD 24a when a warning object to be described later is detected. As a result, the HUD 24a displays a warning icon using a part of the display area. Further, the assisting ECU 10 drives the speaker 24b to alert the driver when transmitting the alerting display information to the HUD 24a.

  The indicator 24a is not limited to the HUD, and a meter display or the like provided on the instrument panel may be used.

  The support ECU 10 includes a microcomputer as a main part. The microcomputer includes a CPU, and a storage device such as a ROM and a RAM, and the CPU realizes various functions by executing a program stored in the ROM.

  Focusing on the function of the microcomputer, the support ECU 10 is roughly divided into a driver visual information providing unit 10a and an alerting unit 10b. In this case, the support ECU 10 is configured by a plurality of ECUs, for example, separately including an ECU including the driver visual information providing unit 10a and an ECU including the alerting unit 10b and combining them. It may be.

  The driver visual information providing unit 10a includes a gaze direction / pupil diameter detection unit 11, a retinal projection image estimation unit 12, a retinal output image estimation unit 13, a local region information amount calculation unit 14, and a personal eye characteristic storage unit 15. It has. The alerting unit 10 b includes an obstacle detection unit 16 and an alerting control unit 17. Each of the functional units 11 to 17 repeatedly performs the following processing at a predetermined short calculation cycle.

  The line-of-sight direction / pupil diameter detection unit 11 analyzes a driver face image representing the face of the driver photographed by the second camera 22, and detects, for example, the movement of the eyeball of the driver and the outline of the pupil. The line-of-sight direction / pupil diameter detection unit 11 detects the movement of the driver's face (head above the neck) based on the detection signal of the gyro / acceleration sensor built in the second camera 21. The line-of-sight direction / pupil diameter detection unit 11 extracts the outline of the pupil from the driver face image, and calculates the center position of the pupil and the pupil diameter from the outline. The line-of-sight direction / pupil diameter detection unit 11 calculates the rotation angle of the eyeball from the relative relationship between the eye contour of the driver (the contour of the eye surrounded by the upper eyelid and the lower eyelid) and the center position of the pupil.

  The line-of-sight direction / pupil diameter detection unit 11 may also be referred to as the driver's line-of-sight direction (line-of-sight position) based on the center position of the driver's pupil, the rotation angle of the driver's eyeball, and the orientation (angle) of the driver's face. Possible). The line-of-sight direction / pupil diameter detection unit 11 outputs information representing the driver's line-of-sight direction and pupil diameter calculated as described above to the retinal projection image estimation unit 12. Various methods are conventionally known for detecting the driver's line-of-sight direction and pupil diameter, and these methods can be arbitrarily employed. In the present embodiment, the second camera 22 is mounted on the driver's head, but may instead be provided fixed to the vehicle. Further, the movement of the driver's head may be detected from a captured image obtained by, for example, photographing the driver's head with a camera fixed to the vehicle instead of the gyro / acceleration sensor.

  The configuration including the second camera 22 and the line-of-sight direction / pupil diameter detection unit 11 corresponds to the line-of-sight detection means of the present invention.

  The retinal projection image estimation unit 12 has a function to estimate the retinal projection image ir (x, y, h, t), which is an image projected on the driver's retina from the external image io (x, y, h, t) by calculation. Part. Here, x and y represent the abscissa position and ordinate position of the video (image), and t represents time. Further, h is a color component h = {r, g, b} of the external image. The external image io (x, y, h, t) represents an image output from the first camera 21. The retinal projection image ir (x, y, h, t) depends on the external image io (x, y, h, t), the driver's line of sight, the driver's pupil diameter, and the driver's eye optical system characteristics. It is determined.

  The eye optical system characteristic is the optical characteristic of the driver's eyeball when the driver does not wear glasses or contact lenses (these are referred to as auxiliary lenses), and the driver's eyeball when the driver wears the auxiliary lens. And an optical characteristic of an optical system including the auxiliary lens.

The retinal projection image estimation unit 12 calculates a retinal projection image ir (x, y, h, t) using a pupil function. The retinal projection images ir (x, y, h, t) are individually calculated for the left and right eyes of the driver.
The pupil function PF (x, y, t) is expressed by the following equation (1).
Here, S (x, y, t) represents the two-dimensional shape of the pupil in the xy plane, and assuming that the two-dimensional shape of the pupil is a circle, the value is 1 when the pupil is located within the pupil radius. If it is outside the circle, the value is set to zero. Λ represents the wavelength of light passing through the driver optical system.
Since the wavelength λ of the light passing through the driver optical system is a mixture of various single wavelengths, it is impossible to calculate all the wavelengths from the image. Therefore, in the present embodiment, a single wavelength is obtained for each pixel using an xy chromaticity diagram (CIExy chromaticity diagram) defined by the International Commission on Illumination (CIE), and the single wavelength is expressed by the above equation (1). Let λ be. In this case, the periphery of the region shown in the CIExy chromaticity diagram is monochromatic light (single wavelength), and the central portion of the region is achromatic. The area around the area shown in the CIExy chromaticity diagram is called a single wavelength curve. In the present embodiment, in the CIExy chromaticity diagram, a straight line connecting a point that is an achromatic color and a point that represents the color coordinates (x, y) of the pixel value (R, G, B) is a single wavelength curve. Find intersecting intersections. The color coordinate (x, y) can be considered as a mixed color of a single wavelength color and an achromatic color at this intersection. Therefore, the wavelength of the monochromatic light specified at this intersection is used as λ in the above equation (1).

  The retinal projection image estimator 12 determines whether the image is within the pupil circle of the external image io (x, y, h, t) based on the information indicating the driver's line-of-sight direction and pupil diameter output from the line-of-sight direction / pupil diameter detector. A range, that is, S (x, y, t) is set.

  In Expression (1), W (x, y) represents the wavefront aberration of the driver's individual eyeball. The characteristics of the driver's individual eye optics are reflected in the wavefront aberration. The wavefront aberration W (x, y) is stored in the personal eye characteristic storage unit 15.

Usually, the wavefront aberration measured by the adaptive optics technique such as the Hartmann-Shack sensor is approximated by a Zernike polynomial such as the following equation (2).
Here, Z is each term of the Zernike polynomial, r (≦ 1) is a polar coordinate radius, and θ is a rotation angle. N is the order in the radial direction, and m is the order in the rotational direction (n ≧ m). As a linear sum of these Zernike terms, the wavefront aberration W (x, y) of the optical system is reconstructed as the following equation (3).
Here, Cn, m is a coefficient of each term of the Zernike polynomial. In the present embodiment, the wavefront aberration does not change with time. However, when the wavefront aberration is changed in accordance with the change in the distance to the object viewed by the driver (change in the focal position). Can be W (x, y, t) including a time element.

  The personal eye characteristic storage unit 15 stores data corresponding to the coefficient Cn, m of the Zernike term as the personal eye characteristic of the driver.

The optical system transfer function OTF (fx, fy, t) representing the eye optical characteristics of the driver can be expressed by the following equation (4) using the pupil function PF (x, y, t).
Here, fx represents the spatial frequency in the x direction, and fy represents the spatial frequency in the y direction. F ₂ represents a two-dimensional Fourier transform function. F ₂ ⁻¹ represents a two-dimensional inverse Fourier transform function.

Each image representing the external image io (x, y, h, t) divided into three RGB components is represented by ioor = io (x, y, r, t) and iog = io (x, y, g, t). ), Iob = io (x, y, b, t), and assuming that these two-dimensional Fourier transforms are IOr, IOg, and IOb, respectively, a retinal projection image ir (projected on the retina after passing through the optical system of the driver) x, y, h, t) can be expressed by the following equation (5).

  In this way, the retinal projection image estimation unit 12 uses the retinal projection expressed by the same RGB value for each frame from the external image io (x, y, h, t) in which each pixel is expressed by the RGB value. The image ir (x, y, h, t) is calculated.

  This retinal projection image estimation unit 12 corresponds to the retinal projection image estimation means of the present invention.

  The personal eye characteristic storage unit 15 is a non-volatile memory and stores the personal eye characteristic of the driver. The user stores data representing the wavefront aberration of the driver's eye optical system (data corresponding to the coefficient Cn, m) measured in an ophthalmic clinic or the like in the personal eye characteristic storage unit 15 via the input unit 23. This personal eye characteristic is an optical characteristic of an optical system including a driver's eyeball and an auxiliary lens when the driver wears an auxiliary lens for driving. In this case, optical characteristics obtained by synthesizing the driver's eyeball and auxiliary lens may be input, or individual optical characteristics thereof may be input.

  The retinal projection image estimation unit 12 reads the data stored in the personal eye characteristic storage unit 15 and calculates the retinal projection image ir (x, y, h, t). When inputting data (individual eye characteristics), a storage medium storing data may be connected to the input unit 23 so that the input unit 23 reads the data, or the user manually operates the input unit 23. Data may be input to the terminal, or data transmitted from an external wireless communication device may be received by the input unit 23, and any form can be adopted.

  The configuration including the input unit 23 and the personal eye characteristic storage unit 15 corresponds to the personal eye characteristic acquisition unit of the present invention.

  FIG. 2 represents the wavefront aberration of an astigmatic eye. In the drawing, a dark portion represents an area having a negative strength, and a thin portion represents an area having a positive strength. In the astigmatic eye, the intensity varies depending on the position of the light receiving surface. On the other hand, in the normal eye, the wavefront aberration is small, and the intensity is constant over almost the entire light receiving surface. FIG. 3 shows a retinal projection image, (a) is a retinal projection image of a normal eye, and (b) is a retinal projection image of an astigmatic eye. As can be seen from this figure, the retinal projection image is unclear in an astigmatic eye having a larger wavefront aberration than that of a normal eye. Even in the case of an eyeball having the same wavefront aberration, when the pupil diameter is large, the retinal projection image is unclear compared to the case where the pupil diameter is small.

  The retinal projection image estimation unit 12 supplies the calculated retinal projection images ir (x, y, h, t) of the left and right eyes to the retinal output image estimation unit 13. The retinal output image estimation unit 13 stores a standard retinal photoreceptor array model for the left and right eyes of a person. This retinal photoreceptor cell arrangement model is configured by arranging three types of cone cells, a red cone, a green cone, and a blue cone, and rod cells according to the anatomical knowledge of the human retina. The

  As shown in FIG. 4A, the more pyramidal cells are located closer to the fovea, the smaller the size is closer to the fovea. The diameter of the cone cell is about 2 μm in the fovea and about 10 μm near the outer 20 deg of the fovea. On the other hand, rod cells are scarcely present near the fovea and are most densely around the outer 20 deg. The diameter of the rod cell is about 1.5 μm when present between the fovea and the outer 40 deg, and about 2.5 μm outside it. In addition, there is a region (blind spot) where no photoreceptor exists in the vicinity of the nasal side 15 deg.

  The retinal photoreceptor cell arrangement model is created in consideration of the spatial arrangement characteristics of human retinal photoreceptor cells. In this retinal photoreceptor cell array model, one of the smallest rod cells is represented by 1 × 1 pixel, and one of the large cone cells located near the outer 20 deg is represented by 5 × 5 pixels. Also, the number of pixels to be expressed in accordance with the size of the human retinal photoreceptor cell is set for rod cells and cone cells at other positions.

If the diameter of the human eyeball is 24 mm, the number of pixels per viewing angle of 1 deg is about 140 [pix / deg]. In the present embodiment, the retinal photoreceptor cell arrangement model is created in the range of viewing angle ± 80 deg. In this case, the retinal photoreceptor cell array model includes 501,760,000 (= 140 pix × 160 deg) ² ) pixels.

  FIG. 4 (b) is a diagram showing the viewing angle range of the retinal photoreceptor array model (gray circular region), and FIG. 4 (c) is a diagram of retinal photoreceptor cells near the fovea in the retinal photoreceptor array model. FIG. 4D is an enlarged view of the arrangement of retinal photoreceptors at a position away from the fovea in the retinal photoreceptor arrangement model. In this example, as shown in FIG. 4 (c), the photoreceptor cells are 1 × 1 pixel blue cones (black in the figure), red cones (dark gray in the figure), It is represented by a green cone (light gray in the figure) and a rod (white in the figure). In the example shown in FIG. 4D as a position away from the fovea, the photoreceptor cells are blue cones slightly smaller than 5 × 5 pixels (the darkest gray portion in the figure), red A cone (the second dark gray part in the figure), a green cone (the third dark gray part in the figure), and a 1 × 1 pixel rod (in the figure) In the lightest gray). One of the lattices shown in FIGS. 4C and 4D corresponds to one pixel.

The delay time from when a human visual cell receives flash light until it reacts is about 30 ms for cone cells and about 70 ms for rod cells. In addition, the response of the photoreceptor cell is maximized for about 100 ms for the cone cell and about 200 ms for the rod cell. Therefore, if the dynamics (impulse response) of a photoreceptor cell is expressed based on such reaction characteristics, it can be expressed as the following expressions (6) and (7).
Here, hcone (t) represents the dynamics of the pyramidal cell, and hrod (t) represents the dynamics of the rod cell. However, hcone (t) and hrod (t) are set to be held at 0 without taking a negative value, as shown in FIG. Also, each integral value is normalized so as to be 1.

In the retinal output image estimation unit 13, the following conversion formula is used to calculate the response of the red cone, the green cone, and the blue cone from the retinal projection image in which each pixel is expressed by an RGB value. . For conversion from RGB values (having values of 0 to 255) of the image into color coordinates XYZ, the standard luminance D65 established by the International Commission on Illumination (CIE) (simulated sunlight with a correlated color temperature of 6504k) The following formula (8) using is used.
Here, R _sRGB , G _sRGB , and B _sRGB are RGB stimulus values in sRGB (standard RGB), and are represented by the following equation (9).
Here, γ represents an index gamma value for correcting the gamma characteristic, and for example, a value such as γ = 2.0, 2.2, 2.4 can be used.
In the above equation (9), R ′, G ′, and B ′ are R ′ = (pixel value R / 255), G ′ = (pixel value G / 255), and B ′ = (pixel value B), respectively. / 255).
Furthermore, Hunt-Pointer proposed in 2002 to convert the color coordinates XYZ calculated by Equation (8) into responses of L, M, and S cones (red, green, and blue cones). The XYZ-LMS conversion formula represented by the following formula (10) is used.

  The retinal photoreceptor cell arrangement model is expressed with the same size (number of pixels) as the retinal projection image. In the retinal photoreceptor cell array model, the RGB value of one pixel of the retinal projection image is converted into an LMS value by the above formula. The corresponding element (pixel) of the retinal photoreceptor array model that receives this is L , M, S cones (red cones, green cones, blue cones) or rods are assigned. In this case, for example, for a pixel to which a red cone is assigned, the LMS value at that pixel is M = 0 and S = 0, and for a pixel to which a green cone is assigned, the LMS at that pixel is set. The values are L = 0 and S = 0, and in the pixel to which the blue cone is assigned, the LMS value at the pixel is L = 0 and M = 0. Thus, of the LMS values converted from the RGB values, responses other than the L, M, and S cones assigned to the elements of the retinal photoreceptor cell array model are set to zero. In this case, for example, in a region expressing one cone with 5 × 5 pixels, the same processing is performed on all the corresponding 25 pixels in the retinal projection image, and the total value thereof is the cone. The response value in the body.

The response dynamics of the L, M, and S cones shown in Expression (6) can be expressed by the following Expression (11).
Here, τ represents a delay time from the current time.

On the other hand, among the retinal projection images expressed by RGB values, the response of the part accepted by the case can be calculated as follows. First, as shown in the following equation (12), an approximate value of the luminance value Rod (x, y, t) is obtained by a weighted average method using NTCS coefficients defined by CCIRrec.601.

The rod response is saturated in bright places. In order to express this saturation characteristic, Rod is assumed to be saturated at 40 at the maximum value 255 for convenience. However, if it does in this way, the pixel value of a housing | casing part will become dark compared with a cone part. Therefore, the rod generates a membrane potential change about 2.5 times that of the cone for the same light stimulus, so the output of Rod saturated at 40 is also multiplied by 2.5. The response dynamics of the housing shown in Expression (7) can also be expressed by Expression (13).

  The retinal output image estimation unit 13 obtains a retinal photoreceptor cell array space sampled image (hereinafter referred to as the retina) representing the time response of the retinal photoreceptor array model with RGB values when the retinal projection image is projected onto the retinal photoreceptor array model. Is called an output image). In this case, the retinal output image estimation unit 13 calculates Ldx which is a value obtained by normalizing Ld so that the maximum value of Ld becomes 255 as a response of the red cone, and the RGB value of the red cone is changed to red. [R = Ldx, G = 0, B = 0] expressed in light and dark. Similarly, the retinal output image estimation unit 13 calculates Mdx which is a value obtained by normalizing Md so that the maximum value of Md becomes 255 as a response of the green cone, and the RGB value of the green cone is changed to green. [R = 0, G = Mdx, B = 0] expressed in light and dark. Similarly, the retinal output image estimation unit 13 calculates Sdx, which is a value obtained by normalizing Sd so that the maximum value of Sd becomes 255 as the response of the blue cone, and converts the RGB value of the blue cone into a blue color. [R = 0, G = 0, B = Sdx] expressed in light and dark.

  This retinal output image estimation unit 13 corresponds to the retinal output image estimation means of the present invention.

  The retinal photoreceptor cell arrangement model may reflect the individual retinal characteristics of the driver. For example, the user stores the retinal characteristics of the left and right eyes of the driver in advance in the personal eye characteristic storage unit 15. For the retinal characteristics, for example, Humphrey perimeter measurement data (measurement data quantified by measuring visual sensitivity at a predetermined viewing angle interval (for example, 6 deg interval)) can be used. In the case of glaucoma patients, there is a portion with low visual sensitivity in a specific region on the retina. Therefore, by storing this retinal characteristic in the personal eye characteristic storage unit 15, it is possible to create a retinal photoreceptor cell arrangement model suitable for a driver having a visual field defect in the retina. In this case, the retinal output image estimation unit 13 uses a visual field cell sensitivity spatial filter that sets a response value (RGB value) to be smaller in a region with lower visual sensitivity based on measurement data of the Humphrey perimeter. What is necessary is just to correct | amend an output image.

  In addition, if the visual field defect region is roughly known without using detailed measurement data, the visual field defect region (eg, upper edge, upper right edge, right edge, lower right edge, lower edge, lower left edge) The left edge, the upper left edge, the center, the upper center, the center upper right, the center right, the center lower right, the center lower, the center lower left, the center left, the center upper left, etc.) The selection may be made over a plurality of areas. Even in this case, the retinal output image estimation unit 13 corrects the retinal output image using the visual field cell sensitivity spatial filter that is set so that the response value in the visual field defect region is small.

  Alternatively, the retinal output image estimation unit 13 may be configured to select a standard retinal photoreceptor cell arrangement model according to the age of the driver. In this case, for example, the retinal output image estimation unit 13 stores a standard retinal photoreceptor cell arrangement model for each age group, and a driver stored in the personal eye characteristic storage unit 15 by the user via the input unit 23. A standard retinal photoreceptor array model according to the age of the driver is selected based on the age of the driver.

  Alternatively, the retinal output image estimation unit 13 may be configured to select a visual field cell sensitivity space filter according to the age of the driver. In this case, for example, the retinal output image estimation unit 13 stores a plurality of visual field cell sensitivity spatial filters for each age group, and the driver's age stored by the user in the personal eye characteristic storage unit 15 via the input unit 23. Based on the above, a visual field cell sensitivity spatial filter is selected according to the age of the driver.

  FIG. 6 shows an example of an external image (a) and a retinal output image (b) calculated from the external image.

The retina output image estimation unit 13 supplies the calculated left and right eye retina output images to the local region information amount calculation unit 14. For each square area A (referred to as local area A) obtained by dividing the retinal output image by a predetermined viewing angle (2 deg in the vertical and horizontal directions in this embodiment), the local area information amount calculation unit 14 A local area information amount H which is an information amount (entry peak) of A is calculated. The local area information amount H is calculated by the following equation (14).
Here, L is the number of gradations, f is a pixel value, and P (f) is a ratio of the pixel value included in the image. FIG. 7 shows an example of the external image local quantity region normalized gradation distribution. In this example, by substituting P (f) into Equation (14), the local area information amount H becomes H = 7.3291. The local area information amount H becomes larger as an image in which the normalized gradation distribution is widened (an image not concentrated on a specific pixel value). This pixel value f corresponds to the RGB value in the retinal output image.

  As shown in FIG. 8, the local area information amount calculation unit 14 slides the local area A (2 deg × 2 deg square area) horizontally by 0.25 deg on the retinal output image, for each slide position. The local area information amount H in the local area A is calculated. The local area information amount calculation unit 14 handles the calculated local area information amount H of each local area A as an image information amount at the center point AO of the local area A. When the calculation (sampling) of the local area information amount H over the entire horizontal direction of the retinal output image is completed, the local area information amount calculation unit 14 performs the same processing at a position where the local area A is slid 0.25 degrees in the vertical direction. repeat. In this way, the local region information amount calculation unit 14 samples the local region information amount H at intervals of 0.25 deg horizontally and vertically over the entire retinal output image.

  An oversight risk map is created by such sampling. As shown in FIG. 9, the oversight risk degree map is obtained by associating each coordinate position (x, y) and local area information amount H in the retinal output image with an interval of a viewing angle of 0.25 deg. It is a dimension map. Each time the retinal output image is switched (the period is also an imaging period of the external image and the calculation period in which the retinal projection image is calculated), the local region information amount calculation unit 14 is the degree of oversight risk corresponding to the retinal output image. Create a map. Therefore, an oversight risk map that changes from moment to moment is created.

  In this case, the local region information amount calculation unit 14 merges the left-eye oversight risk map calculated from the left-eye retinal output image and the right-eye oversight risk map calculated from the right-eye retinal output image. (Overlapping) creates a binocular oversight risk map (hereinafter simply referred to as an oversight risk map). For example, in the state where the left and right retinal output image coordinate positions are matched, the oversight risk map is a local area information amount H of the right eye oversight risk map and local area information of the left eye oversight risk map for each coordinate position. Created by OR information in quantity H. That is, the larger value in the local area information amount H of the left and right eye risk maps is selected and created. Therefore, when the local area information amount H is small for both the left and right eyes, the local area information amount H of the oversight risk degree map is a small value, and when the local area information amount H is large for any one eye, there is an oversight danger. The local area information amount H of the degree map is set to a large value. It should be noted that when the desired local region information amount H is obtained for both the left and right eyes and when the desired local region information amount H is obtained for any one eye, the former local region information amount H is obtained. You may correct | amend so that it may become larger than the latter local region information amount H, or you may correct | amend the latter local region information amount H so that it may become smaller than the former local region information amount H.

  The oversight risk is an index indicating the ease of oversight of an object existing in the driver's field of view, and increases as the local area information amount H decreases. Therefore, the oversight risk map can be paraphrased as a two-dimensional map in which the coordinate position (x, y) in the driver's field of view and the oversight risk are associated with each other at intervals of 0.25 deg horizontally and vertically. it can.

  10 to 13 show examples of an oversight risk map created from an external image (the external image shown in FIG. 6A). In the drawing, the degree of oversight risk is expressed by shading, and the darker the region, the higher the oversight risk is expressed (in other words, the region where the local region information amount H is small). In the examples shown in FIGS. 10 and 11, the oversight risk map is for one eye (left eye) in order to show a blind spot. FIG. 12 is a diagram in which an external image is superimposed on the oversight risk map of FIG. 10 for reference. FIG. 13 shows an example of the binocular oversight risk degree map. For example, FIG. 11 is an oversight risk map for drivers with astigmatic eyes, and there are a wide range of areas with a high risk of oversight compared to the oversight risk map shown in FIG. Note that the oversight risk map shown in FIGS. 10 to 13 is shown in the range of the external image area.

  The local area information amount calculation unit 14 outputs information representing the calculated oversight risk map to the alerting control unit 17. The local area information amount calculation unit 14 corresponds to the local area information amount calculation means of the present invention. The configuration including the retinal projection image estimation unit 12, the retinal output image estimation unit 13, and the local area information amount calculation unit 14 corresponds to the map calculation means of the present invention.

  The alerting control unit 17 alerts the driver based on the information indicating the oversight risk degree map output from the local region information amount calculation unit 14 and the information regarding the obstacle output from the obstacle detection unit 16. The presence / absence of an obstacle to be alerted (judgment target) is determined, and when the alert target is present, the alarm 24 is activated to notify the driver of the presence of the alert target. This alerting control part 17 and the alarm device 24 correspond to the alerting means of the present invention.

  The obstacle detection unit 16 performs image processing on an external image obtained by photographing with the first camera 21, extracts a three-dimensional object from the external image, and determines the type of the three-dimensional object based on the shape and movement of the three-dimensional object. Identify pedestrians, motorcycles, cars, etc. The obstacle detection unit 16 recognizes a three-dimensional object that may collide with the host vehicle as an obstacle, the coordinate position in the external image of the obstacle, and the approach state between the host vehicle and the obstacle (for example, the own vehicle). Information indicating the distance between the vehicle and the obstacle, the approaching speed between the host vehicle and the obstacle, and the like is supplied to the alerting control unit 17. The obstacle detection unit 16 corresponds to the obstacle information acquisition unit of the present invention.

  A driver with normal visual ability can recognize the presence of an obstacle and perform a collision avoidance operation (brake operation and steering wheel operation). However, drivers do not necessarily have normal visual ability and may overlook obstacles. Therefore, the alerting control unit 17 refers to the oversight risk map and limits the degree of approach between the obstacle and the host vehicle only for the obstacle detected in the area (direction) where the driver has a high oversight risk. When the value exceeds the threshold, it is estimated that the driver has overlooked the obstacle, and the obstacle is set as the alert target.

  The oversight risk map corresponds to the coordinates (x, y) in the external image. For this reason, the position of the obstacle detected from the external image can be obtained as a position on the oversight risk map. The alerting control unit 17 extracts the local area information amount H of the area where the obstacle is detected on the oversight risk degree map. In this case, the largest value of the local area information amount H of the central point AO of each local area A in the area where the obstacle is detected is set as the visual information amount of the obstacle (hereinafter referred to as the obstacle visual information amount). Call it). Therefore, when there is at least one local region having a large local region information amount H in the region where an obstacle is detected, the obstacle visual information amount is set to a large value. The smaller the obstacle visual information amount, the higher the risk of oversight.

  For the obstacle whose visual information amount of the obstacle is smaller than the preset information amount threshold, the alerting control unit 17 has a visual field region (hereinafter simply referred to as oversight danger) whose risk of oversight is higher than the preset risk level threshold. It is determined that it is in the area. The alerting control unit 17 calculates the degree of approach between the obstacle and the host vehicle only for the obstacle that is determined to be in the overlooked dangerous area. The degree of approach is a parameter that determines whether or not to call attention, that is, a parameter that determines the timing to call attention.

  For example, the degree of approach may be a collision prediction time that is a time from the current time to a time when the host vehicle and the obstacle are predicted to collide. The predicted collision time can be obtained by dividing the distance between the host vehicle and the obstacle by the relative speed between the host vehicle and the obstacle. Alternatively, as the approach degree, the distance between the host vehicle and the obstacle can be used instead of the collision prediction time.

  The alerting control unit 17 sets the obstacle as an alerting object when the approaching degree between the host vehicle and the obstacle in the overlooked dangerous area becomes larger than a preset approaching degree threshold. Then, the alarm device 24 is operated to notify the driver of the presence of an obstacle (object to be alerted).

  Hereinafter, the alerting control process which the alerting control part 17 implements is demonstrated using a flowchart. FIG. 14 shows an attention control routine executed by the attention control unit 17. The attention calling control routine is repeatedly executed at a predetermined calculation cycle. When this routine is started, the alerting control unit 17 determines whether or not an obstacle is detected by the obstacle detection unit 16 in step S11. If the obstacle is not detected, the alerting control unit 17 once ends this routine.

  The alerting control unit 17 repeats such processing, and when an obstacle is detected (S11: Yes), the processing proceeds to step S12, and whether or not the obstacle is in an overlooked danger area. judge. If the obstacle is not in the overlooked dangerous area, the attention drawing control unit 17 once ends the routine and then repeats the above-described processing. On the other hand, when the obstacle is in the overlooked danger area (S12: Yes), the alerting control unit 17 advances the process to step S13, and calculates the degree of approach G between the host vehicle and the obstacle. In subsequent step S14, it is determined whether or not the approach degree G is larger than the approach degree threshold Gref. When the approach degree G is equal to or smaller than the approach degree threshold Gref (S14: No), the alert controller 17 once ends this routine. When such processing is repeated and the approach degree G becomes greater than the approach degree threshold Gref (S14: Yes), the alert control unit 17 sets the obstacle as an alert target in step S15, and notifies the user. Actuator 24 is activated.

  For example, the alert control unit 17 calculates the collision prediction time as the approach degree G, and when it is determined that the collision prediction time is shorter than the set time (a preset threshold value) (S14: Yes), this obstacle Is set as the alert object. Alternatively, the alerting control unit 17 calculates the distance between the host vehicle and the obstacle as the approach degree G, and determines that the obstacle is less than a set distance (a preset threshold). Set as a reminder. When the obstacle control unit 17 sets an obstacle as a target object for alerting, the alerting control unit 17 operates the alarm 24 to notify the presence of the target object for attention.

  In this way, the alerting control unit 17 sets the approaching degree G to be greater than the approaching degree threshold Gref as an alerting condition, and when the alerting condition is satisfied, the driver is an obstacle (attention target) Inform the existence of

  In this case, the alerting control unit 17 drives the speaker 24b to generate an alerting sound, and transmits alerting display information to the HUD 24a. The alert display information includes a position (referred to as an icon display position) for displaying an alert icon (for example, an arrow icon indicating the alert object), and a direction in which the icon arrow is facing (attention direction). Information) is included.

  The alerting control unit 17 sets the icon display position to a position where the risk of oversight of the driver is smaller than a preset threshold, and directs the attention direction from the icon display position to the alerting object. Set the direction. The HUD 24a projects an alert icon on the windshield based on the alert display information transmitted from the alert controller 17.

  As a result, the driver is alerted by the alert icon projected on the windshield, can recognize the obstacle that has been overlooked, and can perform a collision avoidance operation.

  In order to alert the driver, a voice announcement that informs the direction of the obstacle from the speaker can also be used. Moreover, you may make it combine the display of the icon for alerting | calling, and an audio announcement.

  According to the vehicle driving support system 1 of the present embodiment described above, the driver visual information providing unit 10a is based on the external image, the direction of the driver's line of sight, and the driver's personal eye characteristics. An oversight risk map is created that associates the position of the vehicle with the oversight risk that is an index of the ease of oversight of the driver. Therefore, the easy-to-miss direction in the driver's field of view can be ascertained from the oversight risk map.

  The alerting unit 10b notifies the driver of the presence of the obstacle based on the obstacle detected from the external image and the oversight risk map. Accordingly, it is possible to perform appropriate alerting according to the visual ability of each driver. That is, it is possible to call attention to an obstacle that is estimated to be overlooked by the driver. In particular, in this embodiment, only the obstacles (obstacles whose obstacle visual information amount is smaller than the preset information amount threshold value) are in the visual field region where the risk of oversight is higher than the threshold value. Will be notified. Therefore, it is possible to reduce unnecessary alerts for obstacles recognized by the driver. Thereby, the troublesomeness which a driver feels can be reduced.

  Moreover, when creating the oversight risk map, a retinal projection image corresponding to the driver's personal eye characteristics is estimated, and further, a retinal output image that is a response when this retinal projection image is projected onto the retinal photoreceptor cell array model Is estimated. Then, by calculating the amount of information for each local area of a preset area in the retinal output image, an oversight risk map that associates the position in the driver's field of view with the amount of information is created. The Therefore, it is possible to grasp the external visual situation of each driver using this oversight risk map. As a result, it is possible to alert an obstacle according to the visual environment of each driver. In addition, when a retinal photoreceptor cell array model is set according to the retinal characteristics of each driver, an appropriate oversight risk map can be calculated even for a driver with retinal disease.

<Variation 1 of alerting>
In the above-described embodiment, the driver is notified of the presence of only an obstacle in the visual field region where the risk of oversight is higher than the threshold. In the first modification, the driver is alerted in a situation where the driver does not perform the collision avoidance operation even if the obstacle is not in the field of view where the risk of oversight is higher than the threshold. Has been.

  In the first modification, the alerting control unit 17 is configured to notify the driver of the presence of an obstacle when a preset alerting condition is satisfied, and the visual field region whose risk of oversight is higher than a threshold value The warning conditions for obstacles in the vehicle are easier to notify the driver of the presence of obstacles than the warning conditions for obstacles in the field of view where the risk of oversight is below the threshold. Is set.

  FIG. 15 illustrates an alert control routine according to the first modification. The alerting control unit 17 repeatedly executes the alerting control routine according to the modified example 1 at a predetermined calculation cycle. Hereinafter, processing similar to that of the embodiment will be described with the same step symbols as those of the embodiment in the drawings.

  When this routine is activated, the alerting control unit 17 determines whether or not an obstacle is detected in step S11, and repeats the process until an obstacle is detected. When an obstacle is detected (S11: Yes), the alerting control unit 17 calculates the degree of approach G between the host vehicle and the obstacle in step S13. Subsequently, the alerting control unit 17 determines whether or not the obstacle is in an overlooked dangerous area in step S12. In this step S12, as in the embodiment, it is determined that an obstacle having an obstacle visual information amount smaller than a preset information amount threshold is in an overlooked danger area.

  When the obstacle is in the overlooked danger area (S12: Yes), the alerting control unit 17 determines whether or not the approach degree G is larger than the first approach degree threshold G1ref in step S141, and approaches When the degree G is larger than the first approach degree threshold G1ref (S141: Yes), the process is advanced to step S15, so that this obstacle is set as an alerting target and the alarm 24 is activated. This alerts the driver to the obstacle. If the approach degree G is equal to or less than the first approach degree threshold value G1ref (S142: No), the process of step S15 is skipped and this routine is once ended. Therefore, in this case, the driver is not alerted.

  On the other hand, when it is determined that the obstacle is not in the overlooked dangerous area (S12: No), the alert control unit 17 determines whether or not the approach degree G is greater than the second approach degree threshold G2ref in step S142. Determine whether or not. The second approach degree threshold G2ref is set to a value larger than the first approach degree threshold G1ref (G2ref> G1ref). If the degree of approach G is greater than the second degree of approach threshold G2ref, the alert controller 17 sets the obstacle as an alert target by proceeding to step S15, and sets the alarm 24 Operate. This alerts the driver to the obstacle. If the approach degree G is equal to or smaller than the second approach degree threshold G2ref, the process of step S15 is skipped and the routine is temporarily ended. Therefore, in this case, the driver is not alerted.

  For example, when the collision prediction time is used as the approach degree G, the alerting control unit 17 determines the collision prediction time for the obstacle determined to be in the overlooked dangerous area (the obstacle that is difficult for the driver to visually recognize). When it is determined that the time is shorter than one set time, the obstacle is set as an alert target. On the other hand, for the obstacle that is determined not to be in the overlooked danger area (the obstacle that is easy for the driver to visually recognize), the alerting control unit 17 makes the predicted collision time longer than the second set time (<first set time). When it is determined that the obstacle is short, the obstacle is set as a target object to be alerted. For example, when the distance between the host vehicle and the obstacle is used as the approach degree G, the alert control unit 17 determines an obstacle (an obstacle that is difficult for the driver to visually recognize) that is determined to be in an overlooked dangerous area. When it is determined that the distance between the host vehicle and the obstacle is shorter than the first set distance, the obstacle is set as an alert target. On the other hand, for the obstacle that is determined not to be in the overlooked dangerous area (the obstacle that is easily visible to the driver), the alerting control unit 17 sets the distance between the vehicle and the obstacle to the second set distance (<first When it is determined that the distance is shorter than the set distance), this obstacle is set as an alert target.

  Therefore, an obstacle that is determined to be in the overlooked dangerous area is alerted to the driver earlier than an obstacle that is determined not to be in the overlooked dangerous area. When the driver recognizes the presence of an obstacle and is alerted in a situation where he / she intends to perform a collision avoidance operation on the obstacle, the driver feels bothered by the alert. Therefore, although it is better to call attention early (when the degree of approach is small) in consideration of safety, if attention is given to the driver, it will be done while the collision can be avoided by normal operation. It is better not to. For this reason, the timing of alerting is generally performed at a timing when an obstacle approaches to some extent. However, obstacles may not be recognized depending on the individual visual ability of the driver.

  Therefore, for the obstacles that are determined to be in the overlooked danger area, attention is given to the driver earlier (when the degree of approach is small) compared to the obstacles that are determined not to be in the overlooked danger area. Try to call out. As a result, for obstacles that are estimated to be well recognized, the timing of alerting can be delayed as much as possible. Conversely, for obstacles that cannot be recognized well, the timing of alerting is advanced. It can be set to give the driver enough time to avoid collisions. As a result, it is possible to call attention appropriate to the visual ability of each driver.

<Variation 2 of alerting>
In the first modified example, the alerting manner by the alarm device 24 is the same for the obstacle determined to be in the overlooked dangerous area and the obstacle determined not to be in the overlooked dangerous area. In this modified example 2, the driver's alert mode for an obstacle determined to be in the overlooked dangerous area is compared with the driver's alert mode for an obstacle determined not to be in the overlooked dangerous area. It is configured to increase the alerting power to the.

  FIG. 16 illustrates an alert control routine according to the second modification. The alerting control unit 17 repeatedly executes the alerting control routine according to the modified example 2 at a predetermined calculation cycle. About the process similar to the modification 1, the same step code | symbol as the modification 1 is attached | subjected to a drawing, and only a brief description is given.

  When this routine is activated, the alerting control unit 17 determines whether or not an obstacle is detected in step S11, and repeats the process until an obstacle is detected. When an obstacle is detected (S11: Yes), the alerting control unit 17 calculates the degree of approach G between the host vehicle and the obstacle in step S13. Subsequently, in step S142, the alerting control unit 17 determines whether or not the approach degree G is greater than the second approach degree threshold G2ref. If the degree of approach G is equal to or smaller than the second degree of approach threshold G2ref (S142: No), the alerting control unit 17 advances the process to step S141, and determines whether the degree of approach G is greater than the first degree of approach threshold G1ref. Determine whether or not. The first approach degree threshold G1ref is set to a value smaller than the second approach degree threshold G2ref (G2ref> G1ref), as in the first modification. Therefore, when the host vehicle and the obstacle are approaching, “Yes” is determined in step S141 prior to step S142.

  When the host vehicle and the obstacle are not so close to each other and the degree of approach G is equal to or less than the first approach degree threshold G1ref (S141: No), the alerting control unit 17 once ends this routine. If such a process is repeated and the host vehicle approaches the obstacle and the approach degree G becomes greater than the first approach degree threshold G1ref (S141: Yes), the alert controller 17 performs the process. Proceeding to step S12, it is determined whether or not the obstacle is in an overlooked dangerous area. When the obstacle is in the overlooked dangerous area (S12: Yes), the alerting control unit 17 operates the alarm 24 in the first alerting mode in step S151, and the obstacle is in the overlooked dangerous area. If not (S12: No), in step S152, the alarm 24 is operated in the second alert mode.

  This first alerting mode is set so that the alerting force is greater than that of the second alerting mode. For example, in the first alerting mode, the volume of the notification sound (voice announcement or buzzer sound) of the speaker 24b is set to a larger value than in the second alerting mode. In this case, in the first alerting mode, the notification sound is generated from the speaker 24b at a volume that can reliably notify the driver of the presence of the obstacle. On the other hand, in the second alerting mode, the notification sound is generated from the speaker 24b with a small volume that does not make the driver feel bothersome. Therefore, even if the driver is alerted when the driver recognizes the obstacle, the driver can feel less troublesome.

  When the host vehicle and the obstacle further approach and the approach degree G becomes greater than the second approach degree threshold G2ref (S142: Yes), the alerting control unit 17 advances the process to step S151. Therefore, in this case, the alarm 24 is operated in the first alert mode even for an obstacle that is not in the overlooked dangerous area. This ensures that the driver is aware of the obstacle.

  In this example, the magnitude of the alerting power to the driver is switched depending on the volume of the speaker 24b, but instead of or in addition to that, the alerting power to the driver is displayed depending on the display mode of the display 24a. The size of can also be switched. For example, in the first alert mode, the alert controller 17 sets the size of the alert icon large (this alert icon is referred to as a large icon), and in the second alert mode, The size of the alert icon is set small (this alert icon is called a small icon). The small icon is set to a size that does not make the driver feel bothersome, and the large icon is set to a size that can surely let the driver know the presence of an obstacle.

  Further, the display position of the alert icon can be switched instead of the size of the alert icon. For example, in the first alert mode, the alert controller 17 displays an alert icon at a position that is easy for the driver to visually recognize, and in the second alert mode, the driver does not feel annoyed (first (1) A warning icon is displayed at a position that is less visible than the warning mode. In this case, for example, the display may be switched between the display by the HUD 24a (high alerting power) and the display by the meter display (low alerting power).

<Variation with a separate driver visual information providing device>
In the above embodiment, the support ECU 10 is configured as a single control device, but may be configured such that the function is shared by a plurality of devices. For example, the driving support system for a vehicle may include a driver visual information providing device and a driving support device.

  In this case, the driver visual information providing device includes the driver visual information providing unit 10a, the first camera 21, the second camera 22, and the input unit 23, calculates an oversight risk map, and represents the oversight risk map. Is output. On the other hand, the driving support device includes the alerting unit 10b and the notification device 24, and includes information indicating an oversight risk map output from the driver visual information providing device and an outside world image obtained by photographing with the first camera 21. As described above, based on the obstacle detected from the external image and the oversight risk map, the driver is notified of the presence of the obstacle estimated to be overlooked by the driver via the alarm device 24. .

  As the driver visual information providing device, a mobile terminal device, for example, a smartphone can be used. The user installs an application for calculating an oversight risk map in advance and inputs personal eye characteristics to the mobile terminal device. The driver visual information providing device is fixed at an arbitrary position (for example, on a dashboard) of the vehicle, and is connected to a driving support device that is a vehicle-mounted device (wired communication or short-range wireless communication).

  In this case, at least one of the first camera 21 and the second camera 22 can use a camera built in the driver visual information providing device, or is mounted on an external camera or a vehicle. You can also use a camera.

<Modification of personal eye characteristics>
In the above embodiment, the data representing the wavefront aberration of the driver's eye optical system is used as the personal eye characteristic for estimating the retinal projection image. Instead, the visual acuity measurement result of the driver (for example, Information indicating the measurement result of 5m visual acuity) may be used. In this case, for example, the retinal projection image estimation unit 12 stores a plurality of optical system transfer functions corresponding to the visual acuity measurement values, and the visual acuity input by the driver to the personal eye characteristic storage unit 15 via the input unit 23. An optical system transfer function may be selected according to the measurement result.

  The vehicle driving support system according to the present embodiment and the modification has been described above. However, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the object of the present invention. .

  For example, in the oversight risk map of the present embodiment, the local region information amount H is set at 0.25 deg intervals, but it is not always necessary to set this interval, and according to the calculation load of the microcomputer as appropriate. Can be set. Further, the local area information amount H in the oversight risk degree map only needs to be set in at least two stages.

  Further, in the present embodiment and the modification, whether the area where the obstacle exists is an area where the risk of oversight is higher than the danger threshold (whether the obstacle visual information amount is smaller than the information amount threshold). )), And alerting is performed according to the determination result. However, it is not always necessary to divide the area where the obstacle exists into two levels of high and low risk of oversight. You may make it alert. In this case, for example, an alert condition may be set so that an obstacle present in a region with a high risk of oversight is more likely to be notified to the driver of the presence of the obstacle. In addition, for example, the alerting mode may be set so that the obstacle present in the region with a high degree of oversight risk has a greater alerting power to the driver.

  Further, by simplifying the equations for calculating the retinal projection image, the retinal output image, the local area information amount, and the like, the calculation load of the microcomputer can be reduced. For example, a fixed value can be used for a variable that does not significantly affect the final calculation result of the local area information amount.

  In this embodiment, the retinal projection image estimation unit 12 estimates a retinal projection image based on the driver's personal eye characteristics (eye optical system characteristics). The image may not be estimated, and in this case, any configuration may be used as long as the retinal output image estimation unit 13 estimates the retinal output image based on the driver's personal eye characteristics (retinal characteristics). In this case, the retinal projection image estimation unit 12 may use standard human eye optical system characteristics.

  That is, the personal eye characteristic of the driver stored in the personal eye characteristic storage unit 15 may be an eye optical system characteristic, a retinal characteristic, or both. When the eye optical system characteristics are stored as the driver's individual eye characteristics, the eye optical system characteristics are reflected in the estimation of the retinal projection image, and when the retinal characteristics are stored, the retinal characteristics are When both characteristics are stored in the estimation, the eye optical system characteristics are reflected in the estimation of the retinal projection image, and the retinal characteristics are reflected in the estimation of the retinal output image.

  DESCRIPTION OF SYMBOLS 1 ... Vehicle drive assistance system, 10 ... Support ECU, 10a ... Driver visual information provision part, 10b ... Attention alert part, 11 ... Gaze direction / pupil diameter detection part, 12 ... Retina projection image estimation part, 13 ... Retina output image Estimating unit, 14 ... local area information amount calculating unit, 15 ... individual eye characteristic storage unit, 16 ... obstacle detecting unit, 17 ... attention control unit, 21 ... first camera, 22 ... second camera, 23 ... input unit , 24 ... notification device, 24a ... display device (HUD), 24b ... speaker, A ... local region, H ... local region information amount.

Claims (9)

Photographing means for photographing the outside world ahead of the host vehicle;
Line-of-sight detection means for detecting the direction of the line of sight of the driver of the host vehicle;
Acquire individual eye characteristics that represent at least one of the characteristics of the eyes of the driver, including eyeglasses or contact lenses, and the characteristics of the retina of the driver. Personal eye characteristic acquisition means for
Based on the external image obtained by photographing by the photographing means, the direction of the visual line of the driver detected by the visual line detecting means, and the personal eye characteristics acquired by the personal eye characteristic acquiring means, the visual field of the driver Map calculation means for calculating an oversight risk map that associates a position in the vehicle with an oversight risk that is an index of the ease of oversight of the driver, and represents the oversight risk map calculated by the map calculation means Driver visual information providing device that outputs. The driver visual information providing device according to claim 1,
The personal eye characteristic acquisition means is configured to acquire an eye optical system characteristic of a driver including glasses or a contact lens,
The map calculation means includes:
Based on the external image obtained by photographing by the photographing means, the direction of the driver's line of sight detected by the line of sight detecting means, and the driver's eye optical system characteristics acquired by the personal eye characteristic acquiring means, the driver Retinal projection image estimation means for estimating a retinal projection image that is an image of the external world projected onto the retina of
When a retinal projection image estimated by the retinal projection image estimation means is projected onto the retinal photoreceptor array model using a retinal photoreceptor array model that is a model in which retinal photoreceptors are arrayed, the retinal photoreceptor cell A retinal output image estimation means for estimating a retinal output image that is an image obtained by the reaction of the array model;
By calculating the amount of information for each local area of a preset area in the retinal output image over the entire retinal output image, the position in the driver's field of view and the oversight that increases as the amount of information decreases A driver visual information providing device comprising: a local area information amount calculating means for calculating an oversight risk level map in which risk levels are associated with each other. The driver visual information providing device according to claim 1,
The personal eye characteristic acquisition means is configured to acquire a retina characteristic of a driver;
The map calculation means includes:
A retina that estimates a retinal projection image that is an image of the outside world projected on the retina of the driver, based on the external image obtained by the photographing by the photographing unit and the direction of the driver's visual line detected by the visual line detecting unit. Projection image estimation means;
A model in which retinal photoreceptor cells are arranged, and estimated by the retinal projection image estimation means using a retinal photoreceptor array model set according to the retinal characteristics of the driver acquired by the personal eye characteristics acquisition means A retinal output image estimation means for estimating a retinal output image that is an image obtained by a reaction of the retinal photoreceptor cell arrangement model when a retinal projection image is projected onto the retinal photoreceptor array model;
By calculating the amount of information for each local area of a preset area in the retinal output image over the entire retinal output image, the position in the driver's field of view and the oversight that increases as the amount of information decreases A driver visual information providing device comprising: a local area information amount calculating means for calculating an oversight risk level map in which risk levels are associated with each other. The driver visual information providing device according to claim 1,
The personal eye characteristic acquisition means is configured to acquire an eye optical system characteristic of a driver including glasses or a contact lens, and a retina characteristic of the driver,
The map calculation means includes:
Based on the external image obtained by photographing by the photographing means, the direction of the driver's line of sight detected by the line of sight detecting means, and the driver's eye optical system characteristics acquired by the personal eye characteristic acquiring means, the driver Retinal projection image estimation means for estimating a retinal projection image that is an image of the external world projected onto the retina of
A model in which retinal photoreceptor cells are arranged, and estimated by the retinal projection image estimation means using a retinal photoreceptor array model set according to the retinal characteristics of the driver acquired by the personal eye characteristics acquisition means A retinal output image estimation means for estimating a retinal output image that is an image obtained by a reaction of the retinal photoreceptor cell arrangement model when a retinal projection image is projected onto the retinal photoreceptor array model;
By calculating the amount of information for each local area of a preset area in the retinal output image over the entire retinal output image, the position in the driver's field of view and the oversight that increases as the amount of information decreases A driver visual information providing device comprising: a local area information amount calculating means for calculating an oversight risk level map in which risk levels are associated with each other. The driver visual information providing device according to any one of claims 2 to 4 ,
Equipped with a pupil detection means for detecting the size of the driver's pupil;
The driver visual information providing apparatus configured to estimate the retinal projection image in consideration of the size of the pupil. A vehicle driving support system comprising the driver visual information providing device according to any one of claims 1 to 5 ,
Obstacle information acquisition means for acquiring obstacle information relating to the position of an obstacle existing in front of the host vehicle;
Based on the obstacle information acquired by the obstacle information acquisition means and the information representing the oversight risk map output from the driver visual information providing device, the driver is notified of the presence of the obstacle. A vehicle driving support system comprising: means. The vehicle driving support system according to claim 6 ,
The vehicle driving support system configured to notify the driver of the presence of only an obstacle in the visual field region in which the risk of oversight is higher than a threshold. The vehicle driving support system according to claim 6 ,
The alerting means is configured to notify the driver of the presence of an obstacle when a preset alerting condition is satisfied, and for an obstacle in the visual field region where the risk of oversight is higher than a threshold value The alert condition is set so that it is easier for the driver to be notified of the presence of an obstacle than the alert condition for an obstacle in the visual field where the risk of oversight is less than or equal to the threshold. Vehicle driving support system. The vehicle driving support system according to claim 6 ,
The alerting means for notifying the driver of the presence of an obstacle in the visual field area where the oversight risk is higher than a threshold is in the visual field where the oversight risk is less than or equal to the threshold. A vehicle driving support system configured to increase the alerting power to the driver as compared to the alerting mode of notifying the driver of the presence of an obstacle.