JP2022084266A - Display control device, display device, and image display control method - Google Patents

Abstract

To suppress and prevent an object to be visually recognized from becoming hardly visible due to the emphasis processing of a display image based on a gaze position.SOLUTION: When it is determined that an occupant is gazing a display image and that an object 300 to be viewed is nonexistent, or the object to be viewed exists and the display image does not obstruct the occupant viewing the object to be viewed, a control unit 140 of a display control device executes image emphasis processing control to raise the emphasis level of a portion of the display image within a first prescribed range, and lower the emphasis level of portions of the display image outside the range, with the gaze point of a vehicle as a reference. When it is determined that the occupant is gazing a display image A1b and that the object 300 to be viewed exists and the display image obstructs the occupant viewing the object to be viewed, the control unit 140 executes non-emphasis processing control to reduce the degree of obstruction to viewing the object 300 to be viewed or avoid the obstruction.SELECTED DRAWING: Figure 2

Description

The present invention relates to a display control device mounted on a vehicle such as an automobile, a display device such as a head-up display (HUD) device, and an image display control method.

Patent Document 1 describes an image display device including a level control unit that determines the importance of an information image based on a line-of-sight detection result and changes the enhancement level of the information image according to the determination result.

JP-A-2017-138350 (for example, claims 1 and 2)

The present inventor has clarified the following problems (1) to (3).
(1) For example, an object requiring visual recognition (whether a moving body or a stationary body), a person or animal requiring visual recognition, a real scene requiring visual recognition, etc. (collectively referred to) in the back or vicinity of the displayed image. When there is an "object requiring visibility (or an object requiring attention)"), when highlighting the image being displayed, care must be taken not to interfere with the visibility of the object requiring visibility.
(2) Further, for example, the driver (viewer) may be looking at a part of the displayed image or the other part of the displayed image, and is not the displayed image but the back side. You may be looking at an object that needs to be seen, or you may be looking at the road surface, for example. Therefore, it is preferable to control the highlighting process appropriately in consideration of the gaze point (in other words, the gaze position).
(3) In recent years, it has become possible to display a plurality of display contents over a considerably wide area by using a windshield as a projected member, and each display content is uniformly treated as one image. In control, it can be assumed that appropriate measures cannot be taken.

In the above-mentioned prior art, it is not assumed that there is a target to be visually recognized behind the displayed image, and therefore, there is no disclosure of countermeasures for each of the above-mentioned problems. Therefore, there is room for improvement in the current display control technology. Such a novel problem has been clarified by the present inventor.

One of the objects of the present invention is to suppress or prevent the difficulty in visually recognizing the object to be visually recognized due to the enhancement process of the displayed image based on the gazing point (gaze position).

Other objects of the invention will be apparent to those skilled in the art by reference to the embodiments exemplified below and the best embodiments, as well as the accompanying drawings.

Hereinafter, in order to easily understand the outline of the present invention, embodiments according to the present invention will be illustrated.

In the first aspect, the display control device is
A display control device that is mounted on a vehicle and controls image display.
It has a control unit that controls the visibility of the displayed image based on the information on the gaze point of the occupant of the vehicle and the information on the object to be visually recognized that the occupant of the vehicle should see.
The control unit
It is determined that the occupant is gazing at the displayed image and there is no visible object, or the visible object exists, but the displayed image does not interfere with the visual recognition of the visible object by the occupant. If this is the case, image enhancement processing control including processing for raising the enhancement level of the displayed image within the first predetermined range is performed with the gaze point of the occupant as a reference.
When the occupant is gazing at the displayed image, the visible object is present, and the displayed image is determined to interfere with the visual recognition by the occupant of the visible object.
Do not implement the control to raise the emphasis level,
Or,
Instead of the control for raising the emphasis level, a process for lowering the emphasis level is performed.
Or,
When the process of increasing the emphasis level has already been started, the process of increasing the emphasis level is interrupted, and the process of increasing the emphasis level is started within the range increased by the process of increasing the emphasis level or the process of increasing the emphasis level. Lower the emphasis level so that it is even lower than the previous level,
By doing so, the non-emphasis processing control for reducing the degree of obstruction of visual recognition of the object requiring visual recognition or avoiding the obstruction is implemented.

According to the first aspect, for example, when an occupant (viewer) is looking at the displayed image, but there is a visual object (or an object) within a predetermined range from the gazing point, the image enhancement is performed. , Prioritize the visual recognition of the object to be visually recognized, and carry out the "non-emphasis processing (in other words, the visual facilitation processing)". As a result, the occupant (driver, visual observer) can surely grasp the object to be visually recognized directly or indirectly in the peripheral visual field.

This solves the conventional problem that the image enhancement process can be one of the factors that hinder the visibility of obstacles, people requiring attention, objects, and the like. In other words, obstacles and the like can be confirmed without being overlooked by the occupants, which is beneficial for safe operation, for example.

In addition, the object to be visually recognized can be interpreted in a broad sense. For example, important map elements such as obstacles including stationary objects and moving objects, people and animals, and exits of highways that require confirmation may be included.

Further, as a mode of non-enhancement processing, the enhancement processing is not performed (cancelled), or the enhancement processing is not performed and the enhancement level is lowered instead (for example, the brightness of the displayed image is lowered from the current state). ), Or suspending the already performed enhancement process and lowering the emphasis level within the already increased emphasis range (when restoring the increased emphasis level, it is equivalent to the cancellation process of the emphasis process being performed). It can also include further lowering the emphasis level (eg, lowering the brightness of the displayed image below the current level) beyond the increased enhancement range.

Providing such various non-emphasis processing is the same as adding a new mode of non-emphasis processing that prioritizes the object to be visually recognized, and enriches the display mode and displays an image according to the driving situation. It also contributes to the realization of.

In the second aspect, which is subordinate to the first aspect,
The control unit
When the visual object is present on the back side or in the vicinity of the displayed image when viewed from the occupant, it may be determined that the displayed image hinders the occupant from visually recognizing the visual object.

The second aspect shows a typical example of a case where the occupant interferes with the visual recognition of the object to be visually recognized. When there is an object requiring visual recognition behind the display image that the occupant is looking at, the displayed image may act to hide the object requiring visual recognition (block the visual sense), and therefore it is determined that it interferes with visual perception. Is preferable.

In addition, even when there is a visual object in the vicinity of the display image that the occupant is looking at, for example, the line of sight may be directed toward the display image due to the attractiveness of the display image, and the confirmation of the visual object may be delayed. It is preferable to determine that it is an obstacle.

In the third aspect, which is subordinate to the first or second aspect,
The control unit
The enhancement level may be changed by increasing / decreasing at least one of the brightness and saturation of the displayed image and enlarging / reducing the size of the displayed image.

The third aspect shows a preferred example of adjusting the enhancement level of an image. Changing at least one of the brightness and saturation of the image, or changing the size of the image, is effective, for example, in order to make it easier to confirm the object to be viewed at the back of the image. For example, by lowering the brightness, it becomes easier to confirm the object to be visually recognized on the back side through the image. Further, by reducing the image size, for example, if the visible object on the back side and the image do not overlap in a plan view from the viewpoint of the occupant (viewer), the visible object may be hidden in the image. It disappears and it becomes easy to confirm.

In the fourth aspect, which is subordinate to any one of the first to third aspects,
The control unit
The occupant
When gazing at the object to be visually recognized
Or,
When gazing at the road surface or background of the road on which the vehicle travels, the non-emphasis processing control is performed on the displayed image within the second predetermined range with reference to the gazing point.

In the fourth aspect, even when the occupant is looking at something other than the displayed image, the displayed image may adversely affect the visual inspection of the object to be visually recognized. Therefore, the image in a predetermined range is not emphasized. Will be implemented.

As an example, there is a case where the occupant is looking at an object that requires visual recognition (for example, an anxious forward vehicle, an exit of a highway that can be seen in the distance, etc.). Even in this case, it is conceivable that the displayed image obstructs the visual inspection of the object to be visually recognized. In this case, by performing the non-emphasis processing on the displayed image, the visibility can be improved and the attraction to the displayed image can be reduced, so that the visual inspection can be facilitated.

Another example is when the occupant is looking at the road surface (or background (actual view)) in which the vehicle is in progress. The occupant may be aware of the object to be visually recognized around the road surface (or background) by peripheral vision, and in the vicinity of the recognition range by peripheral vision, the displayed image may be unemphasized to reduce the brightness. By reducing the size, there is an effect that recognition by peripheral vision can be made more reliable.

In the fifth aspect, which is subordinate to any one of the first to fourth aspects,
The control unit
The non-enhancement processing control may be implemented for a part of the displayed image.

In the fifth aspect, when the non-enhancement processing is performed, it is shown that the non-enhancement processing may be performed on a part of the image. For example, it is assumed that the display contents include a vehicle speed display (display A), a speed limit display (display B), and a navigation arrow (display C). These can be regarded as one display image in combination, but in the fifth aspect, for example, when the display C has the greatest effect on the visual inspection of the object to be visually recognized, only the display C is not displayed. It is also permissible to perform highlighting.

Further, for example, the display image is only the navigation arrow (display C), but the size is large, and in particular, when the background color of the navigation arrow is an expansion color (white, red, etc.). By removing the coloring of the background (in other words, deemphasizing a part of the displayed image), the visual sense when looking at the object to be visually recognized in the back may be improved.

In this way, non-emphasis processing may be performed for each display content, and among one display image (one display content), one that is particularly conspicuous or has a large range of obstructing vision is targeted for non-emphasis processing. May be good. By performing the flexible non-enhancement processing, it is possible to facilitate the visual inspection of the object to be visually recognized on the back side or the vicinity without damaging the information of the image being displayed.

In the sixth aspect, which is subordinate to any one of the first to fifth aspects,
The control unit
An image that emphasizes the object to be visually recognized may be displayed, indicating the existence of the object to be visually recognized.

In the sixth aspect, following the non-highlighting process (or together with the non-highlighting process), for example, highlighting by a highlighting frame may be added to the object to be visually recognized. As a result, the occupant can surely visually check and confirm the object to be visually recognized. Therefore, for example, an effect that the occupant can quickly take the necessary safety avoidance action can be obtained.

In the seventh aspect, which is subordinate to any one of the first to sixth aspects,
The control unit
Attention information or warning information about the object to be visually recognized may be notified by voice or display.

In the seventh aspect, the alert information (warning information) may be notified by voice or display following the non-emphasis processing (or together with the non-emphasis processing). In the case of the sixth aspect, the above notification may be performed following (or together with this display) the display of the emphasized image requiring visual recognition. This has the effect that, for example, the occupant can quickly take the necessary safety avoidance action.

In the eighth aspect, which is subordinate to any one of the first to seventh aspects,
The visual object may be at least one of a visible object (whether a moving body or a stationary body), a visible person or animal, and a visible actual scene.

In the eighth aspect, a specific example of the object to be visually recognized is illustrated. The "moving body requiring visibility" is, for example, a preceding vehicle requiring attention, and the "stationary body requiring visibility" is, for example, an obstacle such as a cardboard box falling on the road surface in front. A "person or animal requiring visibility" is a person or animal trying to cross the road ahead. The "real scene requiring visual recognition" is, for example, an exit of a highway, a signboard that serves as a clue to a destination, or the like. By facilitating confirmation of these objects requiring visual recognition, it is advantageous in terms of safety and the convenience of the navigation function is also improved.

In the ninth aspect, which is subordinate to any one of the first to eighth aspects,
The control unit
The mode of the non-enhancement processing control may be different depending on the situation when it is determined that the displayed image interferes with the visual recognition by the occupant of the visual recognition target.

In the ninth aspect, the content of the non-emphasis processing may be different depending on the situation. For example, the content of the non-emphasis processing (for example, the level of the non-emphasis processing, the mode of the non-emphasis processing, etc.) may be different depending on the degree of danger of the object to be visually recognized. Therefore, an appropriate display control that matches the reality is realized.

In the tenth aspect, which is subordinate to any one of the first to ninth aspects,
The control unit
After performing the non-emphasis processing control on the standing image that gives the visual sense to the road surface, the display may be shifted to the display by the road surface superimposed image that gives the visual sense superimposed on the road surface.

According to the tenth aspect, the image of the standing image is deemphasized to facilitate the visual inspection of the object to be visually recognized behind (or in the vicinity of) the image, and the road surface superimposed display provides information without obstructing the view. Can continue to be provided. According to the road surface superimposition display, the visibility is sufficiently secured, and it is possible to confirm the object to be visually recognized and its surroundings on the back side. Therefore, it is also advantageous in terms of safe driving.

In the eleventh aspect, the display device is
An image generator that generates an image, and an image generator
A display unit that displays the image and
An optical system including an optical member that reflects the display light of the image and projects it onto the projected member.
A display control device dependent on any one of the first to tenth aspects, and
Have.

According to the eleventh aspect, it is possible to perform image display control excellent in convenience and safety, taking into consideration not only the information presentation by the displayed image but also the object to be visually recognized in the back side or the vicinity thereof. Therefore, for example, a display device that is highly practical, has high functionality, and is easy to use with sufficient consideration for safety is realized.

In the twelfth aspect, which is subordinate to the eleventh aspect,
The display device may be a head-up display device having an optical member that projects the display light of an image onto the projected member.

According to the twelfth aspect, for example, an easy-to-use head-up display (HUD) device that is highly practical, highly functional, and fully considered for safety is realized.

In the thirteenth aspect, the image display control method is
It is an image display control method.
The visibility of the displayed image is controlled and the visibility of the displayed image is controlled based on the information of the gaze point of the viewer and the information of the object to be visually recognized that the viewer should see.
The viewer is gazing at the displayed image, and the visual object is not present, or the visual object is present, but the displayed image does not interfere with the visual recognition of the visual object. If it is determined that, the enhancement level of the displayed image within the first predetermined range is increased with reference to the gaze point of the viewer, and the enhancement level of the displayed image outside the first predetermined range is decreased. Implement process control and
When the viewer is gazing at the displayed image, the target to be viewed exists, and the displayed image is determined to interfere with the visual recognition of the target to be viewed by the viewer.
Do not implement the control to raise the emphasis level,
Or,
Instead of the control for raising the emphasis level, a process for lowering the emphasis level is performed.
Or,
When the process of increasing the emphasis level has already been started, the process of increasing the emphasis level is interrupted, and the process of increasing the emphasis level is started within the range increased by the process of increasing the emphasis level or the process of increasing the emphasis level. Lower the emphasis level so that it is even lower than the previous level,
By doing so, the non-emphasis processing control for reducing the degree of obstruction of visual recognition of the object requiring visual recognition or avoiding the obstruction is implemented.

According to the twelfth aspect, by introducing the non-enhancement processing, it is possible to suppress and prevent the difficulty in visually recognizing the object to be visually recognized due to the enhancement processing of the displayed image based on the gazing point (gaze position). It becomes possible. This image control method can be applied not only to display control by a HUD device, but also to display control of a display device such as a liquid crystal display, or a projection display device such as a projector.

In addition, providing various non-emphasis processing is the same as adding a new mode of non-emphasis processing that prioritizes the object to be visually recognized, and enriches the display mode and displays images according to the driving situation. It also contributes to the realization.

Those skilled in the art will readily appreciate that the embodiments according to the invention exemplified may be further modified without departing from the spirit of the invention.

FIG. 1A is a diagram showing an example of the configuration of an in-vehicle system (in-vehicle display system) including a display such as a HUD device and a liquid crystal panel, and FIG. 1B is a diagram showing a display example by the HUD device. 2 (A) to 2 (C) are diagrams showing an example of adjusting the brightness of an image according to the viewpoint position when the brightness is lowered by non-enhancement processing. 3 (A) to 3 (C) are diagrams showing an example of adjusting the brightness of an image according to the gazing point when the size (image size) of the image is reduced by non-emphasis processing. FIG. 4 is a diagram showing an example of the configuration of an in-vehicle display system. FIG. 5 is a flowchart showing an example of the control procedure of the control unit (display control device). 6 (A) to 6 (C) are views showing an example of a display image when the enhancement process according to the gazing point is performed (no visual object is required). 7 (A) to 7 (C) are diagrams showing an example of a display image when the enhancement process according to the gazing point is performed (when there is a visual object to be visually recognized). 8 (A) and 8 (B) are diagrams showing an example of the non-emphasis processing carried out following FIG. 7. 9 (A) to 9 (C) are diagrams showing an example of non-emphasis processing when the occupant is looking at the object to be visually recognized. 10 (A) to 10 (C) are diagrams showing an example of non-emphasis processing when the occupant is looking at the road surface. 11 (A) to 11 (D) are diagrams showing an application example of the non-emphasis processing. 12 (A) to 12 (C) are diagrams showing an example of performing non-emphasis processing on a part of an image.

The best embodiments described below have been used to facilitate understanding of the present invention. Accordingly, one of ordinary skill in the art should note that the invention is not unreasonably limited by the embodiments described below.

See FIG. FIG. 1A is a diagram showing an example of the configuration of an in-vehicle system (in-vehicle display system) including a display such as a HUD device and a liquid crystal panel, and FIG. 1B is a diagram showing a display example by the HUD device. The vehicle-mounted system of FIG. 1A includes a display device (including at least one of a display device and a HUD device) as a component.

In FIG. 1A, the direction along the line connecting the left and right eyes of the viewer (in other words, the width direction of the vehicle 1) is the left-right direction (or the lateral direction), and the direction is orthogonal to the left-right direction and the ground or the ground. The direction along the line perpendicular to the surface corresponding to the ground (here, the road surface 6) is the vertical direction (or the height direction), and the direction along the horizontal and vertical directions (vehicle 1). The direction indicating the direction of forward movement and backward movement) is the front-back direction. It can be said that the left-right direction is the X direction, the up-down direction is the Y direction, and the front-back direction is the Z direction. This point is the same in other drawings (FIGS. 6 to 11 and the like).

The display device here is an in-vehicle display device mounted on the vehicle 1. This in-vehicle display device includes a display device (including a control unit 140, a display device control unit 107, and a display device 108 such as a liquid crystal panel), and a head-up display (HUD) device 100 (control unit 140, device main body 120). Includes at least one of). In the example of (A) of FIG. 1, both are mounted on the vehicle 1.

The HUD device 100 is installed in, for example, the dashboard 41. The device main body 120 includes a display 120 such as a liquid crystal panel, a screen (display unit) 160 having a display surface 164, and a curved mirror (concave mirror or the like) 170 having a light reflecting surface 179. The curved mirror (concave mirror or the like) 170 reflects the light from the screen 160 and projects (projects) the display light K onto the windshield (projected member) 2 provided in the vehicle 1. In the windshield (projected member) 2, a part of the display light K is reflected and incident on the viewpoint (eye) A of the viewer (driver, etc.), and the apparent lights E1 to E3 corresponding to each incident light are emitted. By forming an image at the image formation point in front of the viewer, a virtual image (image) is displayed on the virtual image display surface PS. The virtual image display surface 9 is a virtual (apparent) surface set in the real space in front of the occupant (viewer such as a driver) corresponding to the display surface 164 on the screen 160.

Further, as the virtual image display surface 9, for example, an elevation surface a perpendicular to the road surface 6, inclined surfaces b and c inclined with respect to the road surface 6, a road surface overlapping surface d superimposed on the road surface 6, and an occupant (viewer). There is a surface e or the like in which the near side is an elevation surface (including a pseudo elevation surface) and the far side is an inclined surface. In the display using the surface other than the elevation a, the display distance of the virtual image differs depending on the display position on the virtual image display surface, so that 3D display is possible.

The configuration of the above-mentioned HUD device is an example, and the present invention can be applied to various HUD devices such as a 3DHUD device using a lenticular lens and a parallax type 3D HUD device. The type of HUD device does not matter. The present invention can also be applied to the case where AR display is performed using a display (liquid crystal panel or the like 108).

See FIG. 1 (B). In FIG. 1 (B), the same reference numerals are given to the portions common to those in FIG. 1 (A). In the driving scene of FIG. 1B, the vehicle (own vehicle) 1 is traveling on a straight road (road surface 6), and there are no vehicles in front or obstacles to be noted. In front of the driver (viewer) 5 who is an occupant, the vehicle speed display 202 and the speed limit display are displayed in the HUD display area 3 set on the virtual image display surface (for example, the virtual image display surface a in FIG. 1A). Images (virtual images) 7 for a plurality of display contents including 204 and the navigation display 206 are displayed. In the present invention, the displayed image 7 is subjected to "enhancement processing" according to the gazing point and "non-emphasis processing" based on the existence of an object requiring visual recognition such as an obstacle, and the details thereof will be described later. do.

See FIG. 2 (A) to 2 (C) are diagrams showing an example of adjusting the brightness of an image according to the viewpoint position when the brightness is lowered by non-enhancement processing. In FIG. 2A, three quadrangles (rectangles) A1b, A2b, and A3b shown by broken lines are shown, and each quadrangle shows an image (virtual image) for one display content.

Assuming that the predetermined range centered (reference) on the gazing point PS of the occupant (viewer) 5 overlaps with those images (virtual images) A1b, A2b, A3b in the plan view seen from the occupant 5, the displayed image. May adversely affect the visual inspection of the actual scene of the occupant 5. However, in FIG. 1A, since there is no overlap, each image (virtual image) is displayed at a normal luminance level (luminance value L0).

In FIG. 2B, the gazing point PS overlaps with a part of the image (virtual image) A1b. As a result, the image (virtual image) A1b falls within a predetermined range with respect to the gazing point PS. The control unit 140 sets the brightness of the image (virtual image) A1b to L1 (> L0) in order to make it easier to distinguish the image (virtual image) A1b from the other two images (virtual image) A2b and A3b. Raise it. As a result, the image (virtual image) enhancement process (enhancement process by luminance) is performed.

See FIG. 2 (C). In FIG. 2C, there is a box-shaped object (for example, corrugated cardboard: an obstacle 300 in a broad sense) falling on the road behind (or near) the image (virtual image) A1b. Since the obstacle 300 affects the safe driving of the vehicle 1, the occupant (viewer) 5 needs to visually recognize (confirm, etc.) the obstacle 300.

In other words, the obstacle 300 can be said to be a "target to be viewed" (or a "target to be watched" or a "target to be watched"). The "object to be visually recognized" may be not only an obstacle but also a map element such as a vehicle in front that requires attention or an exit (exit) of an expressway. Since these are objects that can be seen by the occupant (viewer) 5, they can also be collectively referred to as "objects of the other party". For example, even when a rear vehicle is imaged by a rear image camera and the image is displayed as a virtual image in the HUD display area 3, the virtual image of the rear vehicle is on the other side when viewed from the occupant 5, so that the object on the other side is the object. Can be.

If the enhancement process is performed in the situation of FIG. 2C, there is a possibility that the obstacle 300 may be difficult to see as a result of the enhancement process. Therefore, in FIG. 2C, the control unit 140 performs “non-emphasis processing”.

In other words, the occupant 5 is gazing at the display image A1b, the visibility target 300 exists, and the display image A1b interferes with the visibility of the display target 300 by the occupant 5 (at least, there is a possibility of that). If it is determined that there is), the control unit 140 does not perform the control to raise the emphasis level (here, the brightness), or instead of the control to raise the emphasis level, performs a process of lowering the emphasis level, or , If the process of increasing the emphasis level has already started, the process of increasing the emphasis level is interrupted, within the range increased by the process of increasing the emphasis level, or before the process of increasing the emphasis level is started. The process of lowering the emphasis level is carried out so that the level is even lower than the level. As a result, the degree of obstruction of visual recognition of the object to be visually recognized 300 can be reduced or can be avoided. Such a display control operation is referred to as non-emphasis processing control.

As a result of performing this non-enhancement processing control, the luminance of the image (virtual image) A1b is adjusted to the luminance value L2. The luminance value L2 is adjusted to a value that satisfies L0 ≦ L2 <L1 or L2 <L0. As a result, the occupant 5 can easily see (visually) the obstacle 300. Therefore, for example, it is possible to quickly take an action such as avoiding the obstacle 300 by operating the steering wheel.

Next, refer to FIG. 3 (A) to 3 (C) are diagrams showing an example of adjusting the brightness of an image according to the gazing point when the size (image size) of the image is reduced by non-emphasis processing. In FIG. 3, the object of emphasis (non-emphasis) is not the brightness but the “image size”. Increasing the image size is to emphasize the image, and reducing the image size is to deemphasize the image.

FIG. 3A is the same as FIG. 2A. In FIG. 3B, the size (size) of the image (virtual image) A3b overlapping with the gazing point PS is increased, and the enhancement process is performed by this.

In FIG. 3C, the obstacle 300 overlaps with the image (virtual image) A3b. Therefore, the image size is reduced. In FIG. 3C, as an example, the image size is smaller than the original image size (the size of the image A3b in FIG. 3A). This non-enhancement process eliminates the image that hinders the visual inspection of the obstacle 300, and facilitates the visual inspection of the obstacle 300 by the occupant 5. The examples of FIGS. 2 and 3 show an outline, and specific examples are shown in FIGS. 6 to 12.

Next, refer to FIG. FIG. 4 is a diagram showing an example of the configuration of an in-vehicle display system. The in-vehicle display system includes a pupil image pickup camera 50, a line-of-sight (or line-of-sight direction) detection unit 51, a viewpoint movement detection unit 52, a gaze position (gaze point) detection unit 53, and a front image pickup camera (or surrounding image pickup camera: concrete). For example, the distance between the destination object detection unit (image processing unit) 55 that detects the destination object (including the object to be watched) and the destination object 55, for example, the principle of triangulation. Includes a distance measuring unit (image processing unit) 56 for detecting using the HUD device 100, and a HUD device 100.

As the front image pickup camera (peripheral image pickup camera), an infrared camera or LIDAR may be used in addition to the visible light camera. Alternatively, by using a plurality of detection means, obstacles and the like may be detected with higher accuracy.

The HUD device 100 includes an information acquisition unit 70, a control unit 140, and a control interface (control I / F) 88. The information acquisition unit 70 acquires various information from the gaze position detection unit 53, the destination object detection unit 55, the distance detection unit (image processing unit) 56, the ECU 60, the communication unit 62, and the like. The acquired information is provided (supplied) to the control unit 140.

The control unit 140 includes a gaze determination unit 72 that determines an object that the occupant is gazing at from the position information of the gaze point, a visibility target determination unit 74 that determines the presence / absence, type, etc. of the visibility target, and the visibility target. The emphasis level and the non-emphasis level are determined based on the positional relationship determination unit 76 between the display image and the object to be visually recognized, which determines whether the display image is behind or near the display image, and the result of the above determination or detection. The emphasis level / non-emphasis level determination unit 78 to be determined, the visibility (brightness / size, etc.) control unit 80, and the display mode change unit (for example, the shape and pattern of the image can be changed) provided as needed. It also includes a function (see FIGS. 11C and 11D) 82 for changing the standing image display to a road surface superimposition display, and an image generation unit 84 (including an image composition unit 86).

The emphasis level / non-enhancement level determination unit 78 determines, for example, the level of enhancement processing to be applied to an image within a predetermined range from the gazing point when the occupant is looking at the displayed image and there is no object to be visually recognized. do. In addition, when the occupant is looking at the displayed image and there is a visible object in the back or near the displayed image, it is determined that the displayed image hinders the occupant from visually recognizing the visible object. Determine the level of non-emphasis processing according to.

In addition, the visibility (brightness / size, etc.) control unit 80 also has a function of determining the presence / absence of highlighting of the object to be visually recognized, the necessity of alerting by sight or voice, and the like. When highlighting or alerting the object to be visually recognized is to be displayed, the control interface (control I / F) 88 is notified to that effect.

The output of the image generation unit 84 is supplied to the display 150 (see also FIG. 1A) in the HUD device 100 via the control interface (control I / F) 88. Further, the control interface (control I / F88) outputs a drive signal of the speaker 170 to give a voice notification when a voice warning (warning) about the object to be visually recognized is given.

Next, refer to FIG. FIG. 5 is a flowchart showing an example of the control procedure of the control unit (display control device). The following procedure is an example and is not limited to this.

The control unit 140 (display control device including the control unit 140) acquires line-of-sight information from the line-of-sight detection unit 51. The line-of-sight information includes information indicating the position of the occupant's gaze point.

Next, in step S2, the information of the destination object including the obstacle (the destination object information) is acquired from the front image pickup camera (for example, the visible light camera) 54. The destination object information is information such as the type, position, and size of the destination object, or, when the destination object is an obstacle, damage to the own vehicle 1 when the destination object comes into contact with the object.

In the description of the following steps, it is assumed that the destination object is an obstacle and the destination object information is obstacle information.

The obstacle can be an obstacle to the running of the vehicle 1, and corresponds to a falling object on the road surface 6, unevenness of the road surface, or the like. For example, when there is a preceding vehicle that requires attention, the vehicle ahead can be an obstacle.

Next, in step S3, it is determined whether or not the occupant is gazing at the image displayed by the head-up display (HUD) device 100. If the result is Yes, the process proceeds to step S4. If No, the processing flow after step S11 is executed.

Step S4 is executed when the determination result is Yes in step S3. In step S4, the enhancement level of the image within a predetermined range (or a certain range) from the gazing point is increased. A predetermined range (certain range) is broadly interpreted and includes various applications. For example, when there are images of a plurality of display contents and the emphasis / non-emphasis is determined for each image of each content, the area occupied by the image of the content to be emphasized / non-emphasis is defined as a predetermined range. Become.

The enhancement level is, for example, the level of luminance or saturation. Increasing the emphasis level makes it easier for the occupant to recognize the content of the image being watched.

Next, in step S5, the enhancement level of the image outside a certain range from the gazing point is lowered. By lowering the emphasis level, it is possible to obtain the effect that the road surface condition, obstacles, etc. on the back side of the image that the occupant is not gazing at can be easily recognized by peripheral vision. In addition, there is an effect that the level of the image with the increased enhancement level is increased and the degree of enhancement is increased.

Next, in step S6, it is determined whether or not there is an obstacle or the like in the back side (or in the vicinity of) the image that the occupant is gazing at. If the result is Yes, the processing flow from step S7 onward is executed. If No, the process returns to the beginning of the processing flow.

Next, in step S7, the process of raising the emphasis level carried out in step S4 is interrupted.

Next, in step S8, the enhancement level of the image within a certain range from the gazing point is lowered according to the obstacle information. For example, when the damage to the vehicle is large when it comes into contact with the vehicle, the brightness and saturation are lowered to make it easier to recognize the obstacle. By the processing of step S7 and step S8, even when the occupant is gazing at the image, the presence of an obstacle in the back side or the vicinity of the image can be easily recognized.

Next, in step S9, the obstacle-enhanced image is displayed according to the obstacle information. The obstacle-enhanced image is, for example, an image of a frame surrounding an obstacle. If the damage to the vehicle when it comes into contact with the obstacle is large, the frame is blinked, and if the damage is small enough to be ignored, the obstacle is blinked. Do not display highlighted images.

Next, in step S10, the speaker is operated according to the obstacle information. For example, if the damage to the vehicle is large when it comes into contact with an obstacle, a voice such as "If you continue to drive, the vehicle will come into contact with the obstacle and the vehicle will be damaged." Is played so that the damage can be ignored. If it is small, the sound will not be played. When step S10 is completed, the process returns to the beginning of the processing flow.

Step S11 is executed when the determination result is No in step S3. In step S11, it is determined whether or not the occupant is gazing at an obstacle behind or near the image. If the result is Yes, step S12 is executed, and if No, the processing flow after step S14 is executed.

Step S12 is executed when the determination result is Yes in step S11. In step S12, as in step S8, by lowering the emphasis level within a certain range from the gazing point according to the obstacle information, the obstacles in the back side or the vicinity of the image can be easily recognized. Since the occupant recognizes the existence of the obstacle (C2) in step S11, the obstacle-enhanced image is not displayed or the speaker is not operated as in steps S9 and S10. When the process of step S11 is completed, the process returns to the beginning of the process flow.

Step S13 is executed when the determination result is No in step S11. In step S13, it is determined whether or not the occupant is gazing at the road surface or the background (actual scene) behind the image. If the result is Yes, the processing flow after step S14 is executed. If No, the process returns to the beginning of the processing flow.

Step S14 is executed when the determination result is Yes in step S13. In step S14, by lowering the emphasis level of the image within a certain range from the gazing point, the road surface and the background on the back side or the vicinity of the image can be easily recognized. When step S15 is completed, the process returns to the beginning of the processing flow.

Next, refer to FIG. 6 (A) to 6 (C) are views showing an example of a display image when the enhancement process according to the gazing point is performed (no visual object is required).

In FIG. 6A, the gaze point PS1 of the occupant 5 is near the distant road surface 6. In this state, for example, it is assumed that the HUD device 100 makes a predetermined display, or the occupant 5 requests a predetermined display.

In this case, in FIG. 6B, an image (composite image) 7 including images of three display contents is displayed. In other words, the vehicle speed display 202, the speed limit display 204, and the navigation display 206 are displayed.

In FIG. 6C, the line of sight of the occupant 5 is moving, and the gazing point PS2 is moving on the speed limit display 204. Along with this, highlighting processing is performed, and control is performed to increase the brightness and saturation of the speed limit display 204 within the predetermined range with respect to the gazing point 204, while the vehicle speed display 202 and navigation outside the predetermined range are performed. The display 206 is controlled to reduce the brightness and saturation.

As in the example of FIG. 6C, it is constant when the occupant (driver, viewer) is looking at the image and there is no object to be viewed (obstacle, etc.) in the back or near the image. A process (an example of image enhancement control) is performed in which the brightness and saturation of an image within a range (predetermined range) are increased and the brightness and saturation of an image outside the range are decreased.

Next, refer to FIG. 7. 7 (A) to 7 (C) are diagrams showing an example of a display image when the enhancement process according to the gazing point is performed (when there is a visual object to be visually recognized). In FIG. 7, the same reference numerals are given to the parts common to those in FIG.

In FIG. 7A, there are box-shaped objects (cardboard and the like) 301 and 302 that are obstacles (objects to be visually recognized) on the road surface 6 on the traveling path of the vehicle 1. The occupant's gaze point PS3 is on the front side of the obstacle 301. In this state, for example, it is assumed that the HUD device 100 makes a predetermined display, or the occupant 5 requests a predetermined display.

In FIG. 7B, the image 7 is displayed, and the gazing point PS4 of the occupant 5 is at a position overlapping the speed limit display 204.

In FIG. 7C, the image enhancement process described in FIG. 6C is performed. As for the obstacle 302, the brightness and saturation of the navigation display 206 are reduced, so that the visual inspection is facilitated. On the other hand, with respect to the obstacle 301 located behind the speed limit display 204, the brightness and saturation of the speed limit display 204 increase, which makes it difficult to visually check. In other words, the enhancement process makes it difficult to visually recognize the obstacle 300.

Next, refer to FIG. 8 (A) and 8 (B) are diagrams showing an example of the non-emphasis processing carried out following FIG. 7.

In FIG. 8A, in order to facilitate the visual inspection of the obstacle (object requiring visual recognition) 301, the “non-emphasis control (or non-emphasis processing)” described above in FIG. 2C is implemented. This facilitates visual inspection (visual recognition) of the obstacle 301.

In FIG. 8B, the obstacle 301 is highlighted by the highlight frame (highlight) 305. In addition, a warning (or warning) voice is output from the speaker 170. In other words, the control unit 140 notifies the attention information (warning information) about the visibility target 301 by voice or display.

This notification process may be performed following (or together with the non-enhanced process) the non-enhanced process of FIG. 8 (A). Further, the display of the emphasis frame 305 and the output of the sound may be performed at the same time, or the emphasis frame 305 may be displayed first. This notification has the effect that, for example, the occupant can quickly take the necessary safety avoidance action.

Next, the non-emphasis processing when the occupant is looking at the object to be visually recognized, or when looking at the road surface or the background (actual scene) will be described. This point can be described with reference to FIGS. 7 and 8 described above, but here, an example different from those of FIGS. 7 and 8 (FIGS. 9 and 10) will be described.

9 (A) to 9 (C) are diagrams showing an example of non-emphasis processing when the occupant is looking at the object to be visually recognized. 10 (A) to 10 (C) are diagrams showing an example of non-emphasis processing when the occupant is looking at the road surface.

In FIG. 9A, the large vehicle 320, which is the object to be visually recognized, is seen diagonally forward. Further, the figure (navigation image) 220 of the arrow for navigation indicating straight ahead is displayed so as to overlap the road surface 6. In other words, the navigation image 220 as a road surface superimposed display is displayed.

In FIG. 9B, the gazing point PS5 is near the rear end of the large vehicle 320. A part of the navigation image 220 exists within a predetermined range Z1 (indicated by a broken line circle in the figure) with the gaze point PS5 as a reference. In this case, the occupant (driver, viewer) 5 may be looking at (recognizing) the vicinity of the tip of the arrow figure 220 as a navigation image in the peripheral visual field, and here, as an example. When the figure 220 of the arrow is a highly attractive expansion color such as red, yellow, and white, it cannot be denied that the image 220 viewed in the peripheral visual field may adversely affect the visual inspection of the visual object 120.

Therefore, in FIG. 9C, the non-emphasis processing (non-emphasis control) is performed on the arrow figure 220 as the navigation image to reduce at least one of the luminance and the saturation. As a result, the above-mentioned adverse effect (for example, the attractiveness of the figure 220 of the arrow) is reduced or suppressed. Therefore, it is advantageous in terms of safe driving.

The driving scene of FIG. 10 (A) is the same as that of FIG. 9 (B). However, in FIG. 10A, the gazing point PS6 is located on the distant road surface 6 (or the actual view in the vicinity thereof). A part (tip portion) of the figure 220 of the arrow as a navigation image exists in the predetermined range Z1 centered on the gaze point (gaze position or gaze area) PS6. In this case, the same adverse effects as in the example of FIG. 10B may occur.

Therefore, in FIG. 10B, the non-emphasis processing (non-emphasis control) is performed on the figure 220 of the arrow to reduce at least one of the luminance and the saturation. As a result, the attractiveness of the arrow figure 220 is reduced or suppressed. Therefore, it is advantageous in terms of safe driving.

Next, refer to FIG. 11 (A) to 11 (D) are diagrams showing an application example of the non-emphasis processing. In the example of FIG. 11, the object to be visually recognized (object to be visually recognized) is an exit (exit: a kind of map information) on a highway.

In FIG. 11A, a small highway exit (exit) 904 can be seen diagonally to the left. In FIG. 11B, a navigation image 906 showing the exit (exit) 904 is displayed. At this time, the gaze point (gaze position, gaze area) PS7 of the occupant 5 overlaps a part of the navigation image 906. If the enhancement process is performed in this state as a general rule, it becomes difficult to visually recognize the exit (exit) 904 on the back side thereof.

Therefore, in FIG. 11C, non-emphasis processing is performed to reduce at least one of the luminance and saturation of the navigation image 906. As a result, the difficulty of visually recognizing the exit (exit) 904 is reduced (suppressed).

In FIG. 11D, the navigation image 907 as a road surface superimposed image (road surface superimposed display) is subsequently displayed (for example, at a timing slightly approaching the exit (exit) 904) to reliably guide the vehicle. ing. Since the road surface is superimposed and displayed, the visibility of the exit (exit) 904, which is the object to be visually recognized, is not obstructed, and good visibility is ensured. Therefore, it is possible to reliably guide to the exit (exit) 904. In addition, since the visibility is sufficiently secured, it is advantageous in terms of safe driving.

Next, refer to FIG. 12 (A) to 12 (C) are diagrams showing an example of performing non-emphasis processing on a part of an image. In the example of FIG. 12, the control unit 140 performs non-emphasis processing control on a part of the displayed image.

In FIG. 12A, a plurality of images 152 to 158 are displayed in the HUD display area 3, and various information is presented. Here, if at least a part of these images 152 to 158 is located within a predetermined range from the gazing point (in other words, duplication occurs), non-emphasis processing is performed on all of these images 152 to 158. It will be.

However, if the brightness and saturation of all the images are uniformly reduced at the same time, the change may give the occupant a sense of discomfort. Therefore, only a part is targeted for non-emphasis processing.

Here, focusing on FIG. 12A, the effect that the background portion (the portion shown with a pattern) of the images 153 and 156 obstructs the visual inspection of the object to be visually recognized on the back side (or the vicinity). It turns out that is high. Therefore, in FIG. 12B, only the background portion is hidden. This makes it easier to see the images on the back side (or near) of those images 153 and 156.

Further, in the example of FIG. 12C, the entire image 153 and image 156 are unemphasized, and further, the outer peripheral portion of the circle of the speed limit display 157 (here, colored with an expanded color such as red or yellow). (Make it a conspicuous part), non-emphasis processing is carried out. This makes it easier to visually recognize the object to be visually recognized on the back side (or near) than in FIG. 12B.

In this way, when the non-enhancement processing is performed, the non-enhancement processing may be performed on a part of the image. For example, it is assumed that the display contents include a vehicle speed display (display A), a speed limit display (display B), and a navigation arrow (display C). These can be regarded as one display image by combining them, but for example, when the display C has the greatest effect on the visual inspection of the object to be visually recognized, the non-emphasis processing is performed only on the display C. Is also acceptable.

Further, for example, when the display image is only the navigation arrow (display C), but the size is large and the background color of the navigation arrow is an expansion color (white, red, etc.), the display image is particularly large. Alternatively, if the coloring of the outer peripheral portion of the circular figure is a bar toning, only the coloring is deleted (in other words, a part of the displayed image is deemphasized), so that visibility in the back is required. It can be expected that the visual sense when looking at the object will be improved.

In this way, non-emphasis processing may be performed for each display content, and among one display image (one display content), one that is particularly conspicuous or has a large range of obstructing vision is targeted for non-emphasis processing. May be good. By performing the flexible non-enhancement processing, it is possible to facilitate the visual inspection of the object to be visually recognized on the back side or the vicinity without damaging the information of the image being displayed.

As described above, according to the present invention, it is possible to suppress or prevent the difficulty in visually recognizing the object to be visually recognized due to the enhancement processing of the displayed image based on the gazing point (gaze position). The present invention also contributes to the enrichment of display modes and the realization of image display according to the operating conditions.

Although the present invention has been described above using the embodiments, the present invention is not limited to this, and various modifications and applications are possible. For example, the control unit 140 may change the mode of the non-emphasis processing depending on the situation in which the displayed image is determined to interfere with the visual recognition by the occupant to be visually recognized. For example, the content of the non-emphasis processing (for example, the level of the non-emphasis processing, the mode of the non-emphasis processing, etc.) may be different depending on the degree of danger of the object to be visually recognized. Therefore, an appropriate display control that matches the reality is realized.

Further, the object to be visually recognized may be at least one of an object requiring visual recognition (whether a moving body or a stationary body), a person or animal requiring visual recognition, and a real scene requiring visual recognition. Here, the "moving body requiring visibility" is, for example, a preceding vehicle requiring attention, and the "stationary body requiring visibility" is, for example, an obstacle such as a cardboard box falling on the road surface in front. A "person or animal requiring visibility" is a person or animal trying to cross the road ahead. The "real scene requiring visual recognition" is, for example, an exit of a highway, a signboard that serves as a clue to a destination, or the like. These are examples and are not limited thereto. By facilitating confirmation of these objects requiring visual recognition, it is advantageous in terms of safety and the convenience of the navigation function is also improved.

In the present specification, the term vehicle can be broadly interpreted as a vehicle. In addition, terms related to navigation (for example, signs, etc.) shall be interpreted in a broad sense in consideration of, for example, the viewpoint of navigation information in a broad sense useful for vehicle operation. Further, the HUD device and the display device (and the display device in a broad sense) include those used as a simulator (for example, an aircraft simulator, a simulator as a game device, etc.).

The present invention is not limited to the above-mentioned exemplary embodiments, and those skilled in the art will be able to easily modify the above-mentioned exemplary embodiments to the extent included in the claims. ..

1 ... Vehicle (own vehicle), 2 ... Windshield (projected member), 3 ... HUD device display area (HUD display area), 6 ... Road surface (equivalent to ground, floor, etc.) (Including surface), 41 ... dashboard, 50 ... pupil imaging camera, 51 ... line-of-sight detection unit, 52 ... viewpoint movement detection unit, 53 ... gaze position (gaze point) detection unit. 54 ... front (or peripheral) image pickup camera, 55 ... destination object detection unit, 56 ... distance detection unit (image processing unit), 72 ... gaze determination unit, 74 ... target to be viewed Judgment unit, 76 ... Positional relationship judgment unit between the displayed image and the object to be visually recognized, 78 ... Emphasis level / non-emphasis level determination unit, 80 ... Visibility (brightness / size, etc.) control unit, 82. ... Display mode changing unit, 84 ... Image generation unit, 86 ... Image composition unit, 100 ... HUD device, 107 ... HUD device display, 107 ... Display control unit, 108 ... Display, 160 ... Screen, 170 ... Curved lens (concave mirror, etc.).

Claims (13)

A display control device that is mounted on a vehicle and controls image display.
It has a control unit that controls the visibility of the displayed image based on the information on the gaze point of the occupant of the vehicle and the information on the object to be visually recognized that the occupant of the vehicle should see.
The control unit
It is determined that the occupant is gazing at the displayed image and there is no visible object, or the visible object exists, but the displayed image does not interfere with the visual recognition of the visible object by the occupant. If this is the case, image enhancement processing control is performed to increase the enhancement level of the displayed image within the first predetermined range and decrease the enhancement level of the displayed image outside the first predetermined range with respect to the gaze point of the occupant. Carry out,
When the occupant is gazing at the displayed image, the visible object is present, and the displayed image is determined to interfere with the visual recognition by the occupant of the visible object.
Do not implement the control to raise the emphasis level,
Or,
Instead of the control for raising the emphasis level, a process for lowering the emphasis level is performed.
Or,
When the process of increasing the emphasis level has already started, the process of increasing the emphasis level is interrupted, and the process of increasing the emphasis level is started within the range increased by the process of increasing the emphasis level or the process of increasing the emphasis level. Lower the emphasis level so that it is even lower than the previous level,
A display control device that reduces the degree of obstruction of visual recognition of the object requiring visual recognition, or performs non-emphasis processing control that avoids obstruction. The control unit
When the visual object is present in the back or near the display image when viewed from the occupant, it is determined that the displayed image hinders the occupant from visually recognizing the visual object.
The display control device according to claim 1. The control unit
The enhancement level is changed by at least one of increasing / decreasing the brightness and saturation of the displayed image and increasing / reducing the size of the displayed image.
The display control device according to claim 1 or 2. The control unit
The occupant
When gazing at the object to be visually recognized
Or,
If you are watching the road surface or background on which the vehicle is traveling,
The non-emphasis processing control is performed on the displayed image within the second predetermined range with respect to the gazing point.
The display control device according to any one of claims 1 to 3. The control unit
The display control device according to any one of claims 1 to 4, wherein the non-enhancement processing control can be performed on a part of the display image. The control unit
Displaying a visible target emphasized image showing the existence of the visible target.
The display control device according to any one of claims 1 to 5. The control unit
Notify the attention information or warning information about the object to be visually recognized by voice or display.
The display control device according to any one of claims 1 to 6. The object to be visually recognized is at least one of an object requiring visual recognition (whether a moving body or a stationary body), a person or animal requiring visual recognition, and a real scene requiring visual recognition.
The display control device according to any one of claims 1 to 7. The control unit
The mode of the non-enhancement processing control is changed depending on the situation when it is determined that the displayed image interferes with the visual recognition by the occupant of the visual recognition target.
The display control device according to any one of claims 1 to 8. The control unit
After performing the non-emphasis processing control on the standing image that gives the visual sense to the road surface, the display shifts to the display by the road surface superimposed image that gives the visual sense superimposed on the road surface.
The display control device according to any one of claims 1 to 9. An image generator that generates an image, and an image generator
A display unit that displays the image and
The display control device according to any one of claims 1 to 10 above, and the display control device.
Display device with. The display device according to claim 11, wherein the display device is a head-up display device having an optical member that projects display light of an image onto a projected member. It is an image display control method.
The visibility of the displayed image is controlled and the visibility of the displayed image is controlled based on the information of the gaze point of the viewer and the information of the object to be visually recognized that the viewer should see.
The viewer is gazing at the displayed image, and the visual object is not present, or the visual object is present, but the displayed image does not interfere with the visual recognition of the visual object. If it is determined that, the enhancement level of the displayed image within the first predetermined range is increased with reference to the gaze point of the viewer, and the enhancement level of the displayed image outside the first predetermined range is decreased. Implement process control and
When the viewer is gazing at the displayed image, the target to be viewed exists, and the displayed image is determined to interfere with the visual recognition of the target to be viewed by the viewer.
Do not implement the control to raise the emphasis level,
Or,
Instead of the control for raising the emphasis level, a process for lowering the emphasis level is performed.
Or,
When the process of increasing the emphasis level has already been started, the process of increasing the emphasis level is interrupted, and the process of increasing the emphasis level is started within the range increased by the process of increasing the emphasis level or the process of increasing the emphasis level. Lower the emphasis level so that it is even lower than the previous level,
A method for controlling an image display, which reduces the degree of obstruction of visual recognition of the object to be visually recognized, or implements non-enhancement processing control for avoiding the obstruction.

Images