JP2018185654A - Head-up display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-up display (HUD) device capable of performing displaying without causing conflict with another displaying by generating a driving support image useful for an occupant, on the basis of a photographic image of an actual view around a vehicle (in front of a vehicle, or the like) that can be visualized by the occupant.SOLUTION: An HUD device comprises a visual line detection part 30 and a display control part 50. The display control part 50 comprises: a gaze position detection part 52 for detecting positions of occupant's gazing points on a photographic image on the basis of visual line information acquired by a visual line detection part 30 and a photographic image acquired by photographing an actual view around a vehicle with photographing means; a display photographic image generation part 56 for determining a gaze region on the basis of the detected positions of the plurality of gazing points, extracting a partial image including the gaze region from the photographic image, and outputting the partial image as a display photographic image; and a display image generation part 58 for generating a display image by arranging the display photographic image in an undrawn area of driving support information inside a display image area corresponding to the display image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a head-up display (HUD) device mounted on a vehicle.

  A head-up display device (hereinafter referred to as a HUD device) is mounted on a vehicle such as a car, and displays an image showing information related to driving, etc., on a projection target member (a windshield having both translucency and reflectivity (front glass) ), A combiner or the like), and a vehicle display device capable of displaying a virtual image superimposed on a real scene in front of the driver (including the foreground and the road surface as a background). Since the necessary information and the like are displayed superimposed on the scenery in front, the driver can efficiently obtain the necessary information without moving a large line of sight while driving the vehicle.

  For example, Patent Documents 1 and 2 disclose a vehicle display device capable of displaying a driving assistance image such as navigation information. In the vehicle display device of Patent Document 1, an image captured by a road-side camera is transmitted by wireless communication. If the roadside camera cannot shoot due to bad weather (for example, fog, rain, and snowfall) when it is sent to and displayed on the screen, a landscape shot in good weather from an external data server that constitutes the preventive safety system It is described that the image receiving apparatus is provided and that the HUD device can be used as a vehicle display device (paragraphs [0002], [0010] to [0012], [0060] to [0065]. ], FIG. 1, FIG. 5, FIG. 6)).

  Further, in the vehicle display device of Patent Document 2, three panels that can display independent images are juxtaposed on the instrument panel of the vehicle, and priority is given to vehicle information, navigation information, route guidance, and the like in the center panel. The structure which displays a driving assistance image with high degree is described (paragraph [0025], FIG. 2, FIG. 7, FIG. 9). In addition, although the image of the blind spot part ahead may be displayed on the center panel (the end of paragraph [0025]), this image is pre-photographed and stored, as in Patent Document 1. It is an image.

JP 2010-257253 A JP 2014-229997 A

  In the conventional vehicle display device, the driving assistance image related to the actual scene seen from the vehicle is an image taken in advance under good weather (Patent Document 1) or the driver (occupant) in case the driving environment deteriorates. These are images of a front blind spot portion (Patent Document 2), all of which are images prepared in advance.

  For example, at the beginning of paragraph [0010] of Patent Document 1, “when driving in good weather, the driver (occupant) can confirm the forward field of view. The front image that the driver sees (can see) while driving is excluded from the driving assistance image.

  Here, for example, the present inventors also have captured images obtained by photographing a real scene around the vehicle (such as the front of the vehicle) that can be seen by the driver (occupant) with an in-vehicle camera (or a camera on the road side), It was recognized that it could be a useful driving assistance image depending on the driving situation.

  However, if an image actually viewed by the driver (occupant) is simply displayed on the HUD device as it is, it may not be a useful driving support image, and thus it is necessary to process the image. In addition, the HUD device originally displays vehicle information and the like (running speed display, warning mark display, etc.) superimposed on the actual background, so in addition to this original display, If an attempt is made to display a driving assistance image based on a captured image obtained by photographing the above, it is expected that competition with the original display will occur, and countermeasures are also required.

  Conventional vehicle display devices that can display driving assistance images have not been studied from the above-mentioned viewpoints recognized by the present inventors, and accordingly, the vehicle surroundings that the driver (occupant) is looking at ( A captured image obtained by photographing a real scene in front of the vehicle with an in-vehicle camera (or a camera installed on the road side) cannot be provided to the driver (occupant) as a useful driving assistance image.

  One object of the present invention is to generate a driving assistance image useful for an occupant based on a captured image of an actual scene around the vehicle (such as in front of the vehicle) that can be visually recognized by the occupant, causing competition with other displays. It is an object of the present invention to provide a head-up display device (HUD device) that can display images without causing them to occur. Other objects of the present invention will become apparent to those skilled in the art by referring to the aspects and best embodiments exemplified below and the accompanying drawings.

  In the following, in order to easily understand the outline of the present invention, embodiments according to the present invention will be exemplified.

In the first aspect, the head-up display device comprises:
A head-up display device that causes a passenger to visually recognize a virtual image of the display image by projecting a display image mounted on the vehicle and displayed on a display surface of an image display unit onto a projection target member provided in the vehicle,
A line-of-sight detection unit for detecting the line of sight of the occupant;
Driving assistance image generated based on driving assistance information related to the operation of the vehicle and a captured image obtained by photographing a real scene around the vehicle that is expected to be visible to a passenger of the vehicle. A display control unit having a function of generating the display image based on the display captured image as
With
The display control unit
A gaze position detection unit that detects the position of the gazing point of the occupant on the captured image based on the line-of-sight information acquired by the line-of-sight detection unit and the captured image obtained by imaging by the imaging unit;
Based on the positions of a plurality of gaze points detected by the gaze position detection unit, determine a gaze area so as to include the plurality of gaze points, and extract a partial image including the gaze area from the captured image, A display captured image generation unit that outputs the extracted partial image as the display captured image;
A display image generation unit that generates the display image by arranging the display captured image in an undrawn area of the driving support information in a display image area corresponding to the display image;
Have

  In the first aspect, an occupant's gaze area is determined based on a plurality of gaze point positions, a partial image is extracted from the captured image (in other words, cut out), the partial image is used as a display captured image, and the displayed captured image is Driving support information (for example, vehicle speed display, warning mark, and arrow navigation indicating that there is a sharp turn ahead) in a display image area corresponding to the display image (in other words, an area on the display surface of the image display unit) Display in an undrawn area (in other words, an empty area or a blank area), thereby generating a display image (in other words, a display image for the HUD device) displayed on the display surface of the image display unit. . Extracting a partial image including the area that the occupant was gazing from (in other words, cutting out) from the captured image and generating a “displayed captured image” as a driving assistance image, unnecessary backgrounds were removed It is possible to provide a useful driving support image, and the display captured image is arranged in an undrawn area (empty area or blank area) in the display image area where the driving support information is not displayed. There is no competition (duplication) between images and driving support information.

  At this time, by enlarging the size of the display captured image so as to adapt to the sky area, it is possible to obtain a display image including an image of a real landscape that is easy to see by making full use of the empty space. For example, an occupant (a gaze point at this time is located outside the projection surface on the projection member of the HUD device) who visually recognizes a person who is assumed to be moving toward the road on a certain far shoulder When the line of sight is moved to the virtual image of the HUD device in order to confirm the current vehicle speed, caution information, etc., the occupant's line of sight momentarily leaves the person. Anxiety that "may come" occurs.

  However, in the first aspect, since the person who the occupant visually recognized first is displayed effectively without being buried in other extra background objects, etc., even when looking at the virtual image of the HUD device, etc. It is possible to appropriately check the person and the place requiring attention. Therefore, the above-mentioned feeling of anxiety is reduced, and if a person jumps out, appropriate measures such as deceleration and sudden braking can be taken quickly. In this way, it is possible to easily provide a driving assistance image useful for the occupant without using a complicated and expensive system for connecting to an external server.

  In addition, when the captured image is displayed as it is, it is not an easy-to-see image because an image unnecessary for the occupant is included, and when the size of the captured image is set to a size that is easily visible to the occupant, the captured image is not displayed. The area occupied by the display area increases, which may hinder the visibility of other display contents (for example, driving support information such as a vehicle speed display and a warning mark). Alternatively, it may be assumed that the captured image cannot be enlarged and displayed in a size that is easy to visually recognize because of interference with other display contents.

  In the first aspect, the captured image is not used as it is, and only the image area useful for the occupant is extracted (cut out) and used, so that an image of an important part is efficiently displayed while suppressing the display area. Therefore, the above problem does not occur.

In a second aspect dependent on the first aspect,
By using at least the detection result of the line-of-sight detection unit, the occupant's line-of-sight has a virtual image external gaze detection unit that determines whether or not the line of sight of the occupant is directed outside the virtual image of the display image,
The gaze position detection unit may detect the position of the gazing point of the occupant when the sight line of the occupant is determined to be outside the virtual image of the display image by the virtual image gaze detection unit. Good.

  In the second aspect, when it is determined that the occupant's line of sight is directed outside the virtual image of the display image (in other words, the occupant's line of sight is out of the projection area of the projection target member), By acquiring the position information of the viewpoint, it is possible to limit the acquired gazing point to only the gazing point about the actual scene that the occupant was actually gazing while reducing the burden on the gaze position detection unit. Become. Therefore, a useful display captured image can be obtained efficiently. As described above, the second aspect is because the line of sight deviates from the actual scene that has been viewed so far when the occupant moves his / her line of sight from the virtual image of the HUD device to the virtual image of the HUD device. A special effect of eliminating anxiety can be obtained. The gazing point seen when the occupant's line of sight deviates from the virtual image of the HUD device is a gazing point with high importance. It becomes possible to obtain the effect.

In a third aspect subordinate to the first aspect,
Whether or not the sight line of the occupant is directed outside the virtual image of the display image or whether or not the virtual image of the driving support information is directed by using at least the detection result of the visual line detection unit A virtual image gaze detection unit for determining whether
The display image generation unit
When it is determined that the sight line of the occupant is directed to the virtual image of the display image or the virtual image of the driving support information, the display captured image is enlarged or maintained at the current size. Placed in the undrawn area of the driving support information,
When it is determined that the sight line of the occupant is directed outside the virtual image of the display image or outside the virtual image of the driving support information by the virtual image outside gaze detection unit, the display captured image is reduced and the driving is performed It may be arranged in an undrawn area of support information, or the display captured image may be hidden.

  In a 3rd aspect, a difference is provided in the size of a display captured image by whether a passenger | crew's eyes | visual_axis is outside or inside the virtual image by a HUD apparatus. In other words, in a situation where the display captured image is considered to be highly important, the display captured image is enlarged (in other words, the size is larger than the initial size obtained by extracting the partial image from the captured image). Alternatively, the visibility is improved (or secured) by maintaining the current size, while the display captured image is less important (including the case where it is considered inconvenient). Is reduced (in other words, smaller than the initial size obtained by extracting the partial image from the captured image) or hidden, thereby suppressing an extra burden on the occupant. .

  In addition, regarding the case classification of “whether the line of sight is outside or inside the virtual image by the HUD device”, whether or not the HUD display image (the entire image including the driving support information and the display captured image) is being viewed. In other words, the viewpoint position may be determined more strictly and the driving assistance information in the display image may be viewed. The fact that the occupant is looking out of the virtual image means that there is a high possibility that the occupant is looking at a real scene such as the front.In this case, if a display captured image of a normal size is displayed, the occupant The situation where the displayed captured image is viewed peripherally (in other words, the display captured image can be seen in the field of view around the occupant, and the same image can be seen twice, so-called double-viewed situation), which may cause confusion instead. For this reason, the displayed captured image is reduced or not displayed.

  Further, when the case is divided depending on whether or not the virtual image of the driving support information is viewed, the following effects can be further obtained. That is, when the viewpoint position of the occupant is within the virtual image of the display image but outside the virtual image of the driving support information, the occupant can see the actual scene behind the projection member that is a translucent member. Is likely to see. Therefore, even in this case, if a display captured image having a normal size is displayed, the displayed captured image may be viewed by the occupant and may cause confusion. By performing the above-described strict case classification, appropriate display control of the displayed captured image can be performed even in such a case. Note that hiding the display captured image may be realized by not arranging the display captured image in the display image area, or may be realized by temporarily hiding the entire display image. .

In a fourth aspect dependent on the first aspect,
The display control unit includes a priority order assigning unit that assigns a priority order according to importance for the components of the driving support information and the display captured image.
The display image generation unit reduces a component having a lower priority than the display captured image among the components of the driving support information, and arranges the display image in an undrawn area of the driving support information. Also good.

  In the fourth aspect, in consideration of the limited size of the display image area (area for displaying an image on the display surface of the image display unit), “each component of driving support information” and “display imaging” Priorities are assigned to “images”, and the constituent elements having lower priority than the displayed captured image among the constituent elements of the driving support information are reduced. Thereby, a predetermined size or more necessary for ensuring visibility for displaying high-priority content (in other words, “displayed captured image” and “component of driving support information with high priority”) is displayed. It is possible to reliably secure a free space having a size of 1 and to display the above important elements more easily.

In a fifth aspect subordinate to the first aspect,
The display control unit includes a priority order assigning unit that assigns a priority order according to the importance for the main constituent elements included in the driving support information and the captured image,
The display captured image generation unit determines a gaze area so as to include the plurality of gaze points based on positions of the plurality of gaze points detected by the gaze position detection unit, and is given by the priority order giving unit Referring to the priority order, a partial image is extracted from the captured image so that at least one of the main constituent elements of the captured image has a higher priority order is included and the gaze area is included. The extracted partial image may be output as the display captured image.

  In the fifth aspect, priorities are given to “components of driving support information and main components included in the captured image”, partial images are extracted from the captured images, and “displayed captured images” are displayed. When generating, an area to be extracted (area to be extracted) is determined in consideration of the priority order.

  In the first aspect described above, the area to be extracted is determined by focusing only on the gazing area of the occupant. In this case, for example, for a person who is driving dimly and is about to jump out onto the road In a situation where the person is not aware and the line of sight is not directed at the person, the image about the person who is about to jump out is not included in the displayed captured image. Cannot display.

  On the other hand, in the fifth aspect, the priority level assigning unit included in the display control unit analyzes the captured image captured by the imaging unit and assigns the priority order to the main components. High priority is certainly given to “the person who is about to jump out”. Therefore, when generating a display captured image, the above-mentioned “person who is about to jump out” can also be obtained by “extracting so as to include at least those with a higher priority and including a gaze area”. It is surely included in the displayed captured image. Therefore, it is possible to alert an occupant with the displayed captured image and prevent an accident in advance. In other words, according to the fifth aspect, it is possible to provide a display captured image as a driving assistance image that is more useful.

In a sixth aspect subordinate to the fifth aspect,
The display image generation unit determines a priority order of the display captured image based on the priority order given by the priority order assigning unit, and includes the display captured image among the components of the driving support information. Those with a priority lower than the priority may be reduced, and the displayed captured image may be arranged in an undrawn area of the driving support information.

  In the sixth aspect, in the fifth aspect, the priority order is referred to, and the processing of reducing the priority order lower than the priority order of the displayed captured image among the components of the driving support information is performed. As described above, in the fifth aspect, the process of “extracting a partial image so that at least a higher priority is included and a gaze area is included” is performed. The size (the size of the displayed captured image) is likely to increase. Therefore, in the sixth aspect, the undrawn area (empty space) is effectively increased by reducing the priority order lower than the priority order of the displayed captured image among the constituent elements of the driving support information. Let Therefore, an area (empty space) having a size equal to or larger than the minimum size for arranging the display captured image can be reliably ensured.

In a seventh aspect dependent on the second or third aspect,
A biological information detection unit for detecting the biological information of the occupant;
The virtual image gaze detection unit may perform the determination based on at least one of a detection result of the visual line detection unit and a detection result of the biological information detection unit.

  In the seventh aspect, in the second and third aspects described above, “a process for determining whether or not the sight line of the passenger is directed outside the virtual image of the display image (or in the third aspect, or The process of determining whether or not the vehicle is directed outside the virtual image of the driving support information) is executed. Depending on the determination result, a process having high influence such as reduction or non-display of the displayed captured image is performed. Therefore, it is necessary to be cautious about the above judgment. Therefore, in the seventh aspect, in the above determination, for example, the above determination is performed in consideration of the detection result of the occupant's biological information in addition to the detection result by the line-of-sight detection unit, and the determination accuracy is further improved (however, It may be based on biological information alone). For example, when “a person who may enter the road is visually recognized on the shoulder”, the occupant's line of sight moves greatly to the side of the road, and the biological information such as the pulse also increases and becomes tense. Is considered to be higher. Therefore, by utilizing biological information such as the pulse, it is possible to estimate that “a driving situation with a higher degree of tension has appeared, and that caused a significant shift in the line of sight.” Then, it is highly possible that the line of sight of the occupant is directed outside the virtual image of the HUD. Therefore, determination with higher accuracy is possible. The biological information is not limited to the pulse, and may be, for example, cerebral blood flow indicating the activity state of the brain, or other biological information.

  Those skilled in the art will readily understand that the illustrated embodiments according to the present invention can be further modified without departing from the spirit of the present invention.

It is a figure which shows the structural example corresponding to the 1st-3rd embodiment of the head-up display (HUD) apparatus according to this invention. 2A and 2B are diagrams illustrating an example of the occupant's viewpoint position acquired by the HUD device of FIG. 1 and an example of a displayed virtual image (a virtual image of a display image). FIG. 3A is a flowchart showing main operation procedures of the HUD device in the first embodiment, and FIG. 3B is an example of image processing and display control corresponding to the flowchart of FIG. FIG. It is a flowchart which shows the main operation | movement procedures of the HUD apparatus in 2nd Embodiment. FIG. 5A is a flowchart showing main operation procedures of the HUD device in the third embodiment, and FIG. 5B is an example of image processing and display control corresponding to the flowchart of FIG. 5A. FIG. 5C is a diagram illustrating an example of the priority order for each component. It is a figure which shows the structural example corresponding to 4th Embodiment of the head-up display (HUD) apparatus according to this invention. FIG. 7A is a flowchart showing main operation procedures of the HUD device in the fourth embodiment, and FIG. 7B is an example of image processing and display control corresponding to the flowchart of FIG. 7A. FIG. 7C is a diagram showing an example of the priority order for each component.

  The best mode described below is used to easily understand the present invention. Accordingly, those skilled in the art should note that the present invention is not unduly limited by the embodiments described below.

  FIG. 1 is a diagram showing a configuration example corresponding to the first to third embodiments of a head-up display (HUD) device according to the present invention. In FIG. 1, the front of the vehicle 1 is the z direction, the direction along the left and right direction of the vehicle (the lateral direction of the vehicle) is the x direction, and the upward direction is the y direction.

  In FIG. 1, the vehicle 1 is a real scene around the vehicle 1 that is expected to be visible to the occupant 200 of the vehicle 1 (mainly forward, but may include left, right, rear, etc. as appropriate). An in-vehicle camera 14 as a photographing means (imaging means) for photographing (imaging) the eye, a line-of-sight camera (pupil photographing camera) 16 as a photographing means for eye-gaze detection for photographing the eyes (pupil) of the occupant 200, and A pulse meter 18 as a biological information measuring unit that detects pulse information as biological information, and an in-vehicle ECU (vehicle side) that collects various vehicle information and transmits it to the HUD device 100 via a communication line (BUS) 5 ECU) 10.

  The in-vehicle camera 14 and the line-of-sight camera 16 take an image at a predetermined cycle (for example, 30 Hz), and the taken image is given a time stamp, and in the detection of the gaze position on the captured image, the gaze at the same time The position and the captured image are associated with each other.

  In the example of FIG. 1, the pulsometer 18 is wound around and mounted on the left arm of the occupant 200 holding the handle (steering wheel) 7. As biological information, heartbeat, cerebral blood flow, and other biological information can be widely used, and are not particularly limited. When measuring the cerebral blood flow, for example, a headband type brain activity measuring device is attached to the head (for example, the forehead portion) of the occupant 200. In the present specification, the biological information can be used as auxiliary information for increasing the accuracy of determination related to the movement of the occupant's eyes (pupil) 2, for example. However, the usage form of the biological information is not limited to this, and the determination of the virtual image gaze by the occupant 200 may be performed based on only the biological information, for example.

  In the example of FIG. 1, the in-vehicle camera 14 is used as an imaging unit that captures the surroundings of the vehicle. However, the present invention is not limited to this, and it is wireless from an imaging camera or a driving support system provided on the road side. The imaging information can be transmitted to the vehicle 1 by communication, and the captured image can be reproduced on the vehicle 1 side. In this case, the imaging camera and the driving support system on the road side function as imaging means. In the example of FIG. 1, the eye (pupil) 2 of the occupant 200 is detected by the line-of-sight camera 16, but the vehicle-mounted camera 14 may also serve as the line-of-sight camera 16, and a camera that shoots the room. (Not shown in FIG. 1) may also serve as the visual line camera 16.

  A head-up display (HUD) device 100 is mounted on the vehicle 1, and a reflecting mirror 85 provided in the optical system 82 displays light L of a display image M displayed on a display surface 83 of an image display unit (for example, a liquid crystal panel) 80. And the display image M is projected on a projection target (a windshield (front glass) having both translucency and reflectivity, a combiner, etc., here referred to as a windshield 3) provided in the vehicle 1. Thus, the occupant 200 is caused to visually recognize the virtual image V of the display image M. Note that various displays can be used as the image display unit 80. In the case of a beam scanning type HUD device, the image display unit 80 is a screen on which an image is drawn, and the front or back surface of the screen is the display surface 83.

  The HUD device 100 includes a line-of-sight detection unit 30, a captured image processing unit 40, a display control unit 50, a biological information detection unit (biological information analysis unit) 60, and a HUD control unit (a driving unit that drives the image display unit 80). 70). The line-of-sight detection unit 30, the captured image processing unit 40, the display control unit 50, and the biological information detection unit (biological information analysis unit) 60 are connected via a communication line (bus) 5, and exchange signals and information with each other. It can be performed.

  The line-of-sight detection unit 30 includes a line-of-sight direction detection unit 32, a line-of-sight vector calculation unit 34, and a storage unit 36. The line-of-sight direction detection unit 32 detects the line-of-sight (line-of-sight direction) of the occupant 200 from the captured image captured by the in-vehicle camera 14 (for example, a stereo camera). The process of detecting the line of sight includes a process of detecting the position of the eyes (pupil) of the occupant 200. The line-of-sight direction detection unit 32 calculates, for example, the position of the pupil (eye position) of the occupant 200 and the position of corneal reflection of the occupant 200 by a stereo vision technique. The line-of-sight vector calculation unit 34 calculates a line-of-sight vector representing the line-of-sight direction of the occupant 200 from the position of the pupil of the occupant 200 and the corneal reflection position. The calculated line-of-sight vector is stored in the storage unit 36.

  The captured image processing unit 40 includes a captured image analysis unit 42 and a storage unit 44. The captured image analysis unit 42 recognizes main objects (buildings such as people, utility poles, white lines, road shoulders, road signs, etc.) included in the captured image by pattern matching processing and stores the analysis results in the storage unit 44. Remember. The analysis result of the captured image is appropriately used in the display image generation process by the HUD device 100.

  The display control unit 50 includes a gaze position detection unit 52, a storage unit 54, a display captured image generation unit 56, and a display image generation unit 58. Further, a virtual image extraneous gaze detection unit 65 is provided as necessary. The virtual image extraneous gaze detection unit 65 detects that the gaze (gaze point) of the occupant 200 is outside the virtual image V by the HUD device 100, and in this detection, the gaze obtained by the gaze detection unit 30 Vectors can be used, and in addition, in order to improve detection accuracy, biological information (such as pulse information) obtained by the biological information measurement unit 18 and the biological information detection unit (biological information analysis unit) 60 can be used in combination ( Later).

  The gaze position detection unit 52 obtains an intersection point between the line-of-sight vector calculated by the line-of-sight vector calculation unit 34 in the line-of-sight detection unit 30 and the xy plane set perpendicular to the road surface ahead of the occupant in the xyz coordinate system. The intersection is detected as the gaze position of the occupant 200 (in other words, the position of the gaze point). In addition, when the gaze position for each of the eyes of the occupant 200 is obtained, the midpoint of the line segment connecting the left and right gaze points can be set as the gaze position (gaze point position). The detected gaze position (gaze point position information) is stored in the storage unit 54.

  The display captured image generation unit 56 determines a gaze area based on the positions (coordinate information) of a plurality of gaze positions (gaze points) detected by the gaze position detection unit 52, and includes the gaze area from the captured image An image is extracted, and the extracted partial image is output as a display captured image as a driving support image.

  The display image generating unit 58 includes, for example, a priority order assigning unit 59, an image size changing unit 61, and an image composition unit 63. The priority level assigning unit 59 and the image size changing unit 61 are provided as necessary (described later). The image composition unit 63 displays driving support information (display area on the display surface 83) corresponding to the display image of the HUD device 100 (image M displayed on the display surface 83 of the image display unit 80). Various information displayed by the HUD device 100, such as a vehicle speed display and a warning symbol, which can be referred to as the original display of the HUD device 100), and display captured images in an undrawn area (empty area or blank area) The image composition processing is executed by arranging them, and thereby the image data of the display image M is generated.

  Reference is now made to FIG. 2A and 2B are diagrams illustrating an example of the occupant's viewpoint position acquired by the HUD device of FIG. 1 and an example of a displayed virtual image (a virtual image of a display image). The example of FIG. 2 (A) shows a state in which the occupant 200 is driving calmly without any particular tension. In FIG. 2 (B), a person who may enter the road is discovered, and the occupant 200 shows a slightly tense state.

  First, reference is made to FIG. In FIG. 2A, the line-of-sight camera 16 is provided on the instrument panel 11 in front of the handle (steering wheel) 7. A projection surface (projection region) 4 on which the display light L of the HUD device 100 is irradiated is set on the windshield 3 of the vehicle 1 (which functions as a projection member of the HUD device 100). The virtual image (virtual image superimposed on the front real scene) V is visually recognized via the (projection area) 4.

  In FIG. 2 (A), the front left sign E2, the right front tree E3, the road edge and the center line E4 are shown as components of the front actual scene that the occupant 200 can visually recognize through the windshield 3. It is described. In FIG. 2A, each of the plurality of squares indicates a gaze position (gaze point) detected by the gaze position detection unit 52, in other words, acquired so far.

  On the projection surface (projection area) 4, the display image HP generated by the display image generation unit 58 is displayed. This display image HP includes “attention mark C0, vehicle speed display (in this case, 60 km / h) C3, arrow mark C2 as a navigation display indicating that there is a turning corner to the right” as driving support information, In addition, the display captured image CP2 is displayed (arranged) in a region (empty region) where the driving support information is not displayed on the projection surface (projection region) 4. Here, the display captured image CP2 is generated by extracting (cutting out) a partial image based on a plurality of past gaze positions (gaze points) of the occupant 200 from a captured image obtained by being captured by the in-vehicle camera 14. It is an image to be.

  In the example of FIG. 2 (A), since various driving support information and the image of the front which had been visually recognized until just before are aggregated in the virtual image V, the occupant 200 confirms the virtual image V. , You can get the information you need at once.

  Next, reference is made to FIG. In FIG. 2 (B), since the person E1 progressing toward the road exists in the left front, the gazing position (gaze point) of the occupant 200 has moved to the person E1 side. As a result, the tree E3 does not enter the gaze area determined by the display captured image generation unit 56. However, since the occupant 200 can see the tree through the windshield 3, a particular problem occurs. Absent. On the other hand, the display captured image CP2 includes the person E1, and a state in which the virtual image of the display captured image CP2 including the person E1 can be visually recognized continues at least for a while. Even when the virtual image V by the HUD device 100 is confirmed, it is possible to pay attention to the movement of the person in front, and the anxiety of the occupant 200 is reduced.

  As described above, the HUD device 100 in FIG. 1 determines a gaze area based on a plurality of current and past gazing point positions (rectangles in FIGS. 2A and 2B) of the occupant 200, and a part from the captured image. An image is extracted (in other words, cut out), the partial image is set as a display captured image CP2, and the display captured image CP2 is displayed on the display image area (in other words, on the display surface 83 of the image display unit 80). 2A and 2B, the driving support information in the area corresponding to the projection area 4 (for example, a vehicle speed display, a warning mark, and an arrow indicating that there is a sharp turn ahead. A display image (in other words, displayed on the display surface 83 of the image display unit 80), which is arranged in an undrawn area (in other words, an empty area or a blank area) of navigation display, etc. If, generates a display image) HP for HUD device.

  Extracting a partial image including an area that the occupant 200 was gazing from (i.e., cutting out) from the captured image, and generating a display captured image as a driving assistance image, which is useful in which unnecessary backgrounds are removed Since the display captured image CP2 is arranged in an undrawn area (empty area or blank area) in the display image area where the driving support information is not displayed, the display captured image CP2 is displayed. There is no competition (duplication) between the image CP2 and the driving support information (C0, C2, C3). At this time, by expanding the size of the displayed captured image so as to adapt to the sky area, it is possible to obtain a display image including an image of a real landscape that is easy to see by making full use of the empty space.

  As described above, in the example of FIG. 2B, the occupant who visually recognizes the person E1 who is assumed to be moving toward the road on a certain distant shoulder (the gaze point at this time is the subject of the HUD device 100). When the line of sight 200 is moved to the virtual image V of the HUD device 100 in order to check the current vehicle speed, caution information, etc., the occupant 200 is positioned on the projection member 3 outside the projection plane (projection area) 4. Since the line of sight leaves the person E1 for a moment, the occupant 200 feels uneasy that "the person may jump out on the road". In FIG. 2B, the movement of the line of sight of the occupant 200 is indicated by a bidirectional arrow.

  However, in the HUD device 100 of FIG. 1, the person E1 visually recognized by the occupant 200 first is effectively displayed without being buried in other extraneous background objects, etc., so that the virtual image V of the HUD device 100 is seen. The person E1 can be confirmed as appropriate when he / she is present. Therefore, the above-mentioned feeling of anxiety is reduced, and in the unlikely event that the person E1 jumps out, appropriate measures such as deceleration and sudden braking can be taken quickly. Therefore, it is possible to easily provide a driving assistance image useful for the occupant 200 without using a complicated and expensive system for connecting to an external server.

  In addition, when the captured image is displayed as it is, it is not an easy-to-see image because an image unnecessary for the occupant 200 is included, and if the size of the captured image is set to a size that is easily visible to the occupant 200, the image is captured. The area occupied by the area necessary for displaying the image becomes large, and the visibility of other display contents (for example, driving support information such as a vehicle speed display and a warning mark) may be hindered. Alternatively, it may be assumed that the captured image cannot be enlarged and displayed in a size that is easy to visually recognize because of interference with other display contents. In the HUD device 100 of FIG. 1, only the image area useful for the occupant 200 is extracted (cut out) and used without using the captured image as it is, so that an image of an important part is efficiently suppressed while suppressing the display area. Therefore, the above problem does not occur.

  In addition, in the case of FIG. 2B, the effect of the HUD device according to the present invention is exerted in the case of FIG. 2B rather than the case of FIG. You may acquire a position. In this case, a virtual image extraneous gaze detection unit 65 is provided in the display control unit 50 of FIG.

  Whether the sight line of the occupant 200 is directed outside the virtual image V of the display image HP in FIGS. 2A and 2B by using the detection result of the sight line detection unit 30 at least. Determine whether or not. In this case, the gaze position detection unit 52 in the display control unit 50 of FIG. 1 determines that the gaze of the occupant 200 is directed outside the virtual image V of the display image HP by the virtual image gaze detection unit 65 ( For example, the gaze position (gaze point position) of the occupant 200 may be detected only in that case.

  As described above, when it is determined that the line of sight of the occupant 200 is directed out of the virtual image V of the display image HP (in other words, the line of sight of the occupant 200 is out of the projection area 4 of the projection member 3). The gazing position of the occupant 200 (information on the position of the gazing point) is acquired, thereby reducing the burden on the gazing position detection unit 52 and the actual scene in which the occupant was actually gazing at the acquired gazing point It is possible to limit to only the gazing point. Therefore, a useful display captured image CP2 can be obtained efficiently.

  As described above, the HUD device 100 of FIG. 1 has been viewed until now when the occupant 200 moves his / her line of sight from the virtual image V of the HUD device 100 to the virtual image V of the HUD device 100. It is possible to obtain an extraordinary effect of dispelling the anxiety caused by the fact that the line of sight deviates from the actual scene. The gazing point seen when the line of sight of the occupant 200 deviates from the virtual image V of the HUD device 100 is a gazing point with high importance, and by efficiently collecting this, the burden on the device is reduced, The above-mentioned special effect can be obtained.

  In addition, the virtual image extraneous gaze detection unit 65 described above is based on the detection result of the visual line detection unit 30 and the detection result (analysis result) of the biological information detection unit 60, “the sight line of the occupant 200 is the virtual image V of the display image HP. You may perform the process which determines whether it is turned outside.

  Based on the determination result, the gaze point detection operation by the gaze position detection unit 52 may be started, or a process having high influence such as reduction or non-display of the displayed captured image may be performed. Therefore, it is necessary to be careful in the above determination. Therefore, in the above determination, the above determination is performed in consideration of the detection result of the biological information of the occupant 200 in addition to the detection result by the line-of-sight detection unit 30, and the determination accuracy is further improved.

  For example, when “a person who may enter the road is visually recognized on the shoulder (in the case of FIG. 2B)”, the sight line of the occupant 200 moves greatly to the side of the road, and the pulse, etc. It is thought that the biometric information of is also in a state where the numerical value rises and tension is higher. Therefore, by utilizing biological information such as the pulse, it is possible to estimate that “a driving situation with a higher degree of tension has appeared, and that caused a significant shift in the line of sight.” Then, there is a high possibility that the sight line of the occupant 200 is directed outside the virtual image V of the HUD 100. Therefore, determination with higher accuracy is possible. The biological information is not limited to the pulse, and may be, for example, a cerebral blood flow indicating a heartbeat or a brain activity status, or may be other biological information.

Hereinafter, a specific description will be given using the first to fourth embodiments.
(First embodiment)
First, referring to FIG. FIG. 3A is a flowchart showing main operation procedures of the HUD device in the first embodiment, and FIG. 3B is an example of image processing and display control corresponding to the flowchart of FIG. FIG. FIG. 3B shows an image example corresponding to steps S1, S3, and S4 in FIG. 3 (A) and 3 (B), the same reference numerals are given to portions common to FIG. 2, and the items described with reference to FIG. 2 also apply to FIGS. 3 (A) and 3 (B). obtain.

  In step S1 of FIG. 3A, the display control unit 50 acquires information on the gaze position of the occupant 200 (indicated by a circle in FIG. 3B) in the captured image CP1 captured by the in-vehicle camera 14. . The viewpoint position indicated by ◯ in FIG. 3 (B) is acquired this time regarding the person E1, in other words, the gaze position information acquired so far indicated by □ in FIG. 2 (B). It was acquired in addition. This viewpoint position information is accumulated in the storage unit 36 every time it is acquired. The accumulated gaze position information can be used to determine a gaze area necessary for generating the display captured image CP2.

  Here, acquisition of the gaze position is not necessarily performed every time the processing loop of steps S1 to S4 is turned. For example, the gaze position detection unit 52 of the display control unit 50 has the gaze position of the occupant 200 other than the virtual image V of the display image HP based on at least one of the detection results of the line-of-sight detection unit 30 or the biological information detection unit 60. Only when it is determined, the gaze position (circle in FIG. 3B) information (coordinate position) may be detected (acquired).

  Thereby, the display control part 50 can acquire only the gaze position ((circle) of FIG. 3 (B)) when the passenger | crew 200 is visually recognizing real scenery other than the virtual image V. FIG. In this case, the area of the display captured image CP <b> 2 is limited to “a visual recognition area when the occupant is visually recognizing the real scenery instead of the virtual image V”, and the occupant 200 becomes a more useful image. This is because the real scenery that is the background of the virtual image V is relatively easy to view even when the virtual image V is viewed, and the occupant 200 may not need to display the virtual image as the display captured image CP2. This is because it is expensive.

  Subsequently, in step S <b> 2, the display captured image generation unit 56 stores a predetermined number (a minimum number that can determine the gaze area of the occupant 200) in the storage unit 36, which is a natural number m of 3 or more. ) It is determined whether or not the information of the above gaze positions is accumulated. By determining the gaze area based on a certain number of gaze position information, the accuracy of the determination is increased. In addition, since the determination is based on a plurality of gaze positions in the past, even if the line of sight moves momentarily, the gaze area is determined in consideration of the most recent viewpoint position. Therefore, it is possible to determine a stable gaze area without being dragged.

  When “accumulated (in the case of Y)” in step S2, in step S3, the display captured image generation unit 56 determines the gaze position information based on a plurality of (a plurality of predetermined number m or more) gaze position information. The gaze area is determined so as to include the partial image, and the partial image is extracted (cut out) according to the gaze area, thereby generating the display captured image CP2. If the predetermined number of gaze position information does not exist, the process returns to step S1.

  Subsequently, in step S4, the image composition unit 63 in the display image generation unit 58 generates the display captured image CP2 as the driving assistance image from the captured image CP1. The display control unit 200 may perform a process of adding a predetermined lifetime to each gaze position information and not adopting (or deleting from the storage unit 36) the gaze position information that has passed the lifetime.

  Subsequently, in step S4, the image composition unit 63 in the display image generation unit 58 converts the display captured image CP2 into the driving support information in the display image area corresponding to the display image HP (as described in FIG. 2 here). , A warning mark C0, a vehicle speed display (60 km / h) C3, and an undrawn area (empty area or blank area) of a navigation mark indicating that there is a turning corner that turns to the right. The captured image CP2 is arranged to generate the display image HP. In step S <b> 5, the display image generation unit 58 outputs the display image HP to the HUD control unit 70.

  By performing such display control, the occupant 200 adjusts the viewpoint to the virtual image V, and even if it becomes difficult to see the real scenery that has been viewed until then, the occupant 200 performs display imaging in the virtual image V. Since the virtual image of the image CP2 is included, the situation of the real landscape can be confirmed as before. Therefore, conventionally, it is possible to eliminate the anxiety that the occupant feels when visually recognizing the virtual image V, such as “it is difficult to detect danger and cannot predict driving unless the actual scenery is viewed”. Furthermore, since the displayed captured image CP2 is limited to an image including the actual scenery that has been watched so far, the HUD device 100 of FIG. 1 is more beneficial to the occupant 200 than when the captured image CP1 is displayed as it is. Image information can be provided. For example, when the captured image CP1 and the displayed captured image CP2 are enlarged and displayed by the same magnification, the individual captured image information (people, objects, lanes, etc.) is displayed in the displayed captured image CP2. It will be displayed larger. Therefore, the visibility of each image information by image enlargement is further improved.

(Second Embodiment)
Reference is now made to FIG. FIG. 4 is a flowchart showing main operation procedures of the HUD device according to the second embodiment. The second embodiment is basically the same as the first embodiment. However, in the second embodiment, when the display control unit 50 arranges the display captured image CP2 in the undrawn area, Depending on whether the gazing position of the occupant 200 is inside or outside the virtual image V, processing such as changing the size of the display captured image CP2 is performed, which is different from the first embodiment. In addition, since the processing such as the size change may affect the reliability of the virtual image display of the HUD device 100, the gaze position of the occupant 200 is inside or outside the virtual image V for the sake of caution. In this determination, not only the line-of-sight information but also the biological information is taken into consideration.

  First, in step S1, the gaze position detection unit 52 of the display control unit 50 detects (acquires) the gaze position (gaze point) on the captured image, stores the information in the storage unit 54, and the biological information detection unit 60. Detects (acquires) biological information. Next, the display captured image generation unit 56 determines whether or not the gaze position information is present in the storage unit 54 by a predetermined number (m described above) or more. Next, the display captured image generation unit 56 extracts (cuts out) the partial image from the captured image CP1 to generate the display captured image CP2.

  Next, in step S <b> 4, the virtual image extraneous gaze detection unit 65 provided in the display control unit 50 of FIG. 1 reads the gaze position information from the storage unit 54 and the biometric information sent from the biometric information detection unit 60. Based on the received information, it is determined whether or not the occupant 200 is gazing at the virtual image V (that is, the viewpoint is in the virtual image).

  When Y is determined in step S4, in step S5, the image size changing unit 61 included in the display captured image generation unit 63 enlarges the size of the display captured image CP2 (or maintains the current size), and the driving support information ( It is placed in the undrawn area of the vehicle speed display. Thereby, visibility can be improved.

  Further, in the case of N in step S4, in step S6, the image size changing unit 61 included in the display captured image generation unit 63 reduces the size of the display captured image CP2 as compared with the case where the virtual image V is being watched. The driving assistance information (vehicle speed information, etc.) is arranged in an undrawn area, or the display captured image CP2 is not arranged in the non-drawing area (in other words, not displayed). Accordingly, it is possible to reduce or eliminate the uncomfortable feeling that occurs when the occupant 200 sees the displayed captured image CP2 in the vicinity. Subsequently, in step S <b> 7, the display control unit 50 outputs the generated display image HP to the HUD control unit 70.

  Here, in step S4, it is determined whether “the passenger is gazing at the virtual image V”. The determination may be based on only the line-of-sight information, or the biological information may be used as additional determination material only when the determination is difficult only with the line-of-sight information.

  For example, when the gazing position of the occupant 200 straddles the virtual image V and the actual background, estimation of the degree of tension based on the amount of increase in the pulse, estimation of the psychological state based on the load information on the brain, etc. It can be. Specifically, for example, if the pulse or brain load is equal to or less than a predetermined value, the occupant 200 relaxes and looks far away through the windshield 3 (that is, the occupant is looking at a place other than the virtual image V). If it can be determined that the possibility is high and the load is equal to or greater than a predetermined value, the occupant 200 is in some state of tension and is thus looking at the subject that is causing the tension, In this case, it is highly likely that an object in front is being watched. In this case, in order to collect necessary information, it is impossible to deny the possibility that the occupant 200 looks at the virtual image V. Can do.

  Further, in step S4, not “whether the line of sight of the occupant 200 is directed to the virtual image V or the virtual image V” but “virtual image of driving support information (speed display, etc.) in the virtual image V”. It may be determined whether or not it is directed to the outside of the virtual image of the driving support information (speed display or the like). In this case, the display control unit 200 can reduce or hide the captured image even when the occupant 200 looks around the vehicle 1 through the background portion (virtual image region other than the driving support information) in the virtual image V. Therefore, it is possible to reduce a sense of incongruity that occurs when the occupant 200 gazes at the virtual image V while viewing the displayed captured image CP2 in the vicinity.

  Thus, in the second embodiment, a difference is provided in the size of the display captured image CP2 depending on whether the line of sight of the occupant 200 is outside or inside the virtual image V by the HUD device 100. In other words, in a situation where the display captured image CP2 is considered to be highly important, the display captured image CP2 is enlarged (in other words, a size larger than the initial size obtained by extracting the partial image from the captured image CP1). In other words, it is possible to improve (or ensure) the visibility by maintaining the current size, while the importance of the displayed captured image is relatively low (including the case where it is considered to cause inconvenience) Reduction of the displayed captured image CP2 (in other words, a size smaller than the initial size obtained by extracting the partial image from the captured image) or non-display causes an extra burden on the occupant. To suppress that.

  In addition, regarding the case classification “whether the line of sight is outside or inside the virtual image V by the HUD device 100”, as described above, the display image of the HUD device 100 (the entire image including the driving support information and the display captured image is included). The image may be divided in terms of whether or not the user is viewing the HP, and moreover, the position of the viewpoint is more strictly determined, and the case is divided depending on whether or not the driving support information is viewed in the display image. Also good.

  The fact that the occupant 200 is looking out of the virtual image V means that there is a high possibility that the occupant 200 is looking at a real scene such as the front. In this case, if the display captured image CP2 of a normal size is displayed. In this situation, the occupant 200 sees the displayed captured image CP2 in the periphery (in other words, the display captured image CP2 is seen in the field of view around the occupant 200, and the same two images are seen, so-called double projection). However, since there is a possibility of confusion, the displayed captured image is reduced or not displayed.

  Further, when the case is divided depending on whether or not the virtual image of the driving support information is viewed, the following effects can be further obtained. That is, when the viewpoint position of the occupant is within the virtual image of the display image but outside the virtual image of the driving support information, the occupant is behind the projection member 3 that is a translucent member. This means that there is a high possibility of seeing a real scene. Therefore, even in this case, if a display captured image having a normal size is displayed, the displayed captured image may be viewed by the occupant and may cause confusion. By performing the above-described strict case classification, appropriate display control of the displayed captured image can be performed even in such a case. Note that hiding the display captured image may be realized by not arranging the display captured image in the display image area, or may be realized by temporarily hiding the entire display image. .

(Third embodiment)
Reference is now made to FIG. FIG. 5A is a flowchart showing main operation procedures of the HUD device in the third embodiment, and FIG. 5B is an example of image processing and display control corresponding to the flowchart of FIG. 5A. FIG. 5C is a diagram illustrating an example of the priority order for each component. FIG. 4B shows an image example corresponding to each of steps S4 to S6 in FIG. Further, in FIGS. 5A to 5C, the same reference numerals are given to the portions common to the above-described drawings (particularly FIGS. 2 and 3), and the matters described with reference to FIGS. 5 can also be applied in FIG.

  In the present embodiment, the display control unit 50 displays the displayed captured image CP2, and driving support information (here, a warning mark C1, an arrow mark C2 as a navigation display indicating that there is a turning corner to the right, a vehicle speed display ( 60 km / h) C3)) is given priority, and components having lower priority than the displayed captured image CP2 (C2 and C3 are applicable as can be seen from FIG. 5C) Is reduced to increase the empty space, and low-priority components are kept inconspicuous, and the display captured image CP2 is arranged in the undrawn area to generate the display image HP. As a result, information with high priority can be presented to the occupant 200 in an easy-to-read manner.

  This will be specifically described below. As shown in FIG. 5C, the priority order is highest for the component C1, and the priority order decreases in the order of CP2, C2, and C3. The assignment of priority is performed by a priority assignment unit 59 included in the display image generation unit 58 of FIG.

  In FIG. 5A, steps S1 to S3 are the same as steps S1 to S3 in FIG. In step S4, the priority level assigning unit 59 gives priority levels to the components (C1 to C3) of the driving support information and the display captured image CP2. The result of assigning priorities is as shown in FIG.

  Next, in step S5, it is understood that the priority is lower than the priority of the display captured image CP2 among the components (C1 to C3) of the driving support information (from FIG. 5C, C2 and C3. ). In step S6, the generated display captured image CP2 is arranged in an undrawn area of the driving support information (C1 to C3), and this is set as a final display image HP. In step S <b> 7, the display control unit 50 outputs the display image HP to the HUD control unit 70.

  Here, a specific example of assigning priorities to the display captured image CP2 and the driving support information (C1 to C3) will be described. Different methods of assigning priorities are used for the display captured image CP2 and the driving support information (C1 to C3).

  For the driving support information (C1 to C3), a score (hereinafter referred to as priority) for determining a priority order set in advance for each component (sometimes simply referred to as “element” or “display content”). Score).

  On the other hand, the priority order of the displayed captured image CP2 is determined in consideration of the analysis result of the captured image CP1. That is, the captured image analysis unit 42 included in the captured image processing unit 40 in FIG. 1 extracts the detection object (various elements included in the captured image) and the information accompanying the extracted object from the captured image CP1, and stores the storage unit. 44. The priority level assigning unit 59 included in the display image generation unit of the display control unit 50 reads the detection object and the accompanying information from the storage unit 44, and assigns a priority score to each detection object based on the information. Then, the priority score of the display captured image CP2 is obtained by comprehensively considering the given points. Thereby, the priority order of the display captured image CP2 is obtained.

  Note that the detected object corresponds to, for example, a person E1, a label E2, and a center line E4 as shown in the upper diagram of FIG. The information accompanying the detected objects includes the category, the inter-vehicle distance, and the size of each detected object. The priority score for each detected object is calculated by multiplying the associated information by a coefficient. Of course, by setting the coefficient of certain information to 0, some information may not be included in the calculation of the priority score. Further, the priority score of the displayed captured image CP2 may be calculated by the display control unit 50 using an arbitrary algorithm. For example, if the priority score of the displayed captured image CP2 is the highest score of the priority score of each detected object, the most driving assistance information elements can be reduced in step S5 of FIG. As a result, the arrangement area of the display captured image CP2 can be enlarged. However, if the highest score is given, there is a risk that even a component having a relatively high priority may be reduced. Therefore, it is preferable that the priority score of the display captured image CP2 is carefully determined so that the visibility is balanced in both the component to be reduced and the display captured image CP2, for example, the average value of the extracts. . Note that the reduction rate may be arbitrarily determined according to the priority score for each component to be reduced.

(Fourth embodiment)
In the fourth embodiment, priority is given to “components (C1 to C3) of driving assistance information and main components (E1 to E4) included in the captured image CP1”, and the captured image CP1. When the partial image is extracted from the image and the display captured image CP2 is generated, the region to be extracted (region to be extracted) is determined in consideration of the priority order.

  First, referring to FIG. FIG. 6 is a diagram showing a configuration example corresponding to the fourth embodiment of the HUD device according to the present invention. Although the configuration is generally the same as that in FIG. 1, in FIG. 6, a priority level assigning unit 53 is provided in the display control unit 50 in addition to the configuration in FIG. Other configurations are the same as those in FIG. In FIG. 6, the information flow is indicated by a thick dotted line for easy understanding of the information input to the priority level assigning unit 53.

  Reference is now made to FIG. FIG. 7A is a flowchart showing main operation procedures of the HUD device in the fourth embodiment, and FIG. 7B is an example of image processing and display control corresponding to the flowchart of FIG. 7A. FIG. 7C is a diagram showing an example of the priority order for each component. Note that FIG. 7B shows an image example corresponding to each of steps S3, S4, S6, and S7 in FIG. 7A. Further, in FIGS. 7A to 7C, the same reference numerals are given to the portions common to the above-described drawings (FIGS. 2, 3, and 5), and FIGS. The matter can also be applied in FIG.

  Steps S1 and S2 in FIG. 7A are the same as steps S1 and S2 in FIGS. 3, 4, and 5A. In step S3, the priority level assigning unit 53 in FIG. 6 gives priority levels to the respective components of the driving support information (C1 to C3) and the respective components (E1 to E4) of the captured image CP1. The method of assigning priorities is the same as that described in the third embodiment. The result of assigning priorities is as shown in FIG. The component C1 has the highest priority, and the priority decreases in the order of E1, E2, C2, C3, E4, and E3.

  In step S4, a captured image is extracted based on the gaze position and the priority order of each component, thereby generating a display captured image CP3.

  In other words, the display captured image generation unit 56 determines the gaze area so as to include a plurality of gaze points based on the positions of the plurality of gaze points detected by the gaze position detection unit 52, and the priority level assigning unit 53. Referring to the priority order given by (FIG. 7C), so that at least one of the main constituent elements (E1 to E4) of the captured image CP1 with the higher priority order is included, and A partial image is extracted from the captured image CP1 so as to include the gaze region, and the extracted partial image is output as a display captured image CP3.

  Here, the image corresponding to step S4 in FIG. 7B is referred to. The occupant 200 looks only on the road, and does not look at the person E1 who appears toward the left and moves toward the road. In other words, the passenger 200 is not aware of the person E1. In this case, according to the first to third embodiments, the partial image extracted from the captured image CP1 is determined by the plurality of gaze position information of the occupant 200. Therefore, although the display captured image CP2 is generated, the display captured image CP2 does not include the person E1 who needs attention. However, in the case of the present embodiment, since the second highest priority is assigned to the person E1 (FIG. 7C), this priority order is also taken into consideration and the partial image is included so as to include the person E1. Therefore, a display captured image CP3 including the person E1 is generated.

  Next, in step S5, the priority order assigning unit 59 of the display image generating unit 58 determines the priority order of the generated display captured image CP3 based on the priority order given in step S3. Here, the priority of the display captured image CP3 is lower than the component C1 and higher than the components C2 and C3. In other words, regarding the priority, the relationship of C1> CP3> C2, C3 is established. .

  Next, in step S6, the image size changing unit 61 of the display image generating unit 58 has a priority lower than the priority of the display captured image CP3 among the components (C1 to C3) of the driving support information ( Here, the size of C2, C3) is reduced. Note that the current size of the component C1 is maintained (see the image corresponding to step S6 in FIG. 7B).

  Next, in step S7, the image composition unit 63 of the display image generation unit 58 converts the display captured image CP3 into an undrawn area (empty area or blank area) of the driving support information (C1 to C3) in the display image area. ) To generate a final display image HP. In step S <b> 8, the display image HP is output to the HUD control unit 70.

  As described above, in the fourth embodiment, priorities are given to “the components (C1 to C3) of the driving support information and the main components (E1 to E4) included in the captured image CP1” in advance. When a partial image is extracted from the captured image CP1 and a display captured image is generated, a region to be extracted (region to be extracted) is determined in consideration of the priority order, thereby generating a display captured image CP3.

  In the first embodiment, since the region to be extracted is determined by focusing only on the gazing region of the occupant 200, for example, to the person E1 who is driving dimly and is about to jump out onto the road In a situation where the person is not aware and the line of sight is not directed to the person (an image corresponding to step S4 in FIG. 7B), an image of the person E1 who is about to jump out is included in the display captured image CP2. Therefore, it is not possible to display an image of the person E1 who is a highly important object.

  On the other hand, in the fourth embodiment, the priority level assigning unit 53 newly included in the display control unit 50 analyzes the captured image CP1 captured by the imaging unit (the in-vehicle camera 14), and the main components Since priority is given to (E1 to E4), a high priority (second priority in FIG. 7C) is definitely given to the above-mentioned “person E1 who is about to jump out”. become.

  Therefore, when generating the display captured image, the above-mentioned “person E1 who is about to jump out” is extracted by “extracting so as to include at least those having higher priority and including a gaze area”. Is certainly included in the displayed captured image CP3. Therefore, it is possible to alert the occupant with the display captured image CP3 (including the virtual image V of the display image HP) and prevent an accident in advance. In other words, according to the present embodiment, it is possible to provide a display captured image CP3 as a driving assistance image that is more useful.

  Further, as described above, in the present embodiment, the process of “extracting a partial image so as to include at least a higher priority and a gaze area” is performed. It is considered that the size of the display captured image CP3 that is a partial image tends to increase. Therefore, in the present embodiment, among the components (C1 to C3) of the driving support information, those having a lower priority order (C2, C3) than the priority order of the display captured image CP3 are reduced, so that no drawing is performed. The area (margin area) is effectively increased. Therefore, an area (empty space) larger than the minimum size for arranging the display captured image CP3 can be surely secured.

  As mentioned above, although this invention was demonstrated based on some embodiment, this invention is not limited to these, A various deformation | transformation and application are possible. For example, in the above-described embodiment, an in-vehicle camera images a front scene, but an image of a scene captured by a road-side camera or a driving assistance image captured by a camera of a driving assistance system is captured in a vehicle. In this case, the same effect can be obtained.

  Further, in the above-described embodiment, an object (such as a person moving toward the road) that should be noted mainly in front of the vehicle traveling on the road (preferably interpreted in a broad sense including diagonally or laterally). Although an example in which is present has been described, the present invention is not limited to this.

  For example, the present invention can also be applied when the vehicle is moving backward (back). For example, when an occupant is backing a vehicle, the rear is photographed (captured) with an in-vehicle camera mounted on the vehicle, the image is analyzed on the vehicle side, and navigation by voice or the like is performed. In some cases, navigation information may be displayed.

  While the vehicle is moving backward while visually recognizing the rear, when the occupant sees the front virtual image for a moment in order to confirm the display of the HUD device, the rear real scene cannot be visually recognized. However, when the present invention is used, the virtual image of the HUD device can include a virtual image of a rear real scene image (in other words, a “displayed captured image”). With this virtual image, the rear real scene can be confirmed almost in real time, so that even if an unexpected situation such as a sudden jump out of a person occurs, an appropriate countermeasure can be taken. Thus, the present invention can be widely used in response to various operating situations.

  The present invention is not limited to the above-described exemplary embodiments, and those skilled in the art will be able to easily modify the above-described exemplary embodiments to the extent included in the claims. .

  DESCRIPTION OF SYMBOLS 1 ... Vehicle, 2 ... Viewpoint (eyes, pupil), 3 ... Windshield (member to be projected in a broad sense), 4 ... Projection area, 10 ... In-vehicle ECU, 11 ... Instrument panel, 14 ... vehicle-mounted camera, 16 ... gaze camera, 18 ... biological information measurement unit (pulse meter, etc.), 30 ... gaze detection unit, 32 ... gaze direction detection unit, 34 ... gaze vector calculation unit, 36 ... storage unit, 40 ... captured image processing unit, 42 ... captured image analysis unit, 44 ... storage unit, 50 ... display control unit, 52 ... gaze position detection unit, 53 ... priority assignment unit, 54 ... storage unit, 56 ... display captured image generation unit, 58 ... display image generation unit, 59 ... priority assignment , 60... Biometric information detection unit, 61... Image size changing unit, 63. Display captured image generation unit, 63... Image composition unit, 65... Virtual image gaze detection unit, 70... Image display unit, 70. ... Image display unit, 82 ... Optical system, 83 ... Display surface, 85 ... Reflector, 100 ... HUD device, 200 ... Crew, 200 ... Biological information measuring unit, C0, C1 ... warning mark, C2 ... arrow mark, C3 ... vehicle speed display, CP1 ... captured image, CP2, CP3 ... displayed captured image, E1 ... human, E2,. -Sign, E3 ... Tree, E4 ... Center line, road edge, etc. HP ... Display image, L ... Display light, M ... Display image, V ... Virtual image.

Claims (7)

A head-up display device that causes a passenger to visually recognize a virtual image of the display image by projecting a display image mounted on the vehicle and displayed on a display surface of an image display unit onto a projection target member provided in the vehicle,
A line-of-sight detection unit for detecting the line of sight of the occupant;
Driving assistance image generated based on driving assistance information related to the operation of the vehicle and a captured image obtained by photographing a real scene around the vehicle that is expected to be visible to a passenger of the vehicle. A display control unit having a function of generating the display image based on the display captured image, and
With
The display control unit
A gaze position detection unit that detects the position of the gazing point of the occupant on the captured image based on the line-of-sight information acquired by the line-of-sight detection unit and the captured image obtained by imaging by the imaging unit;
Based on the positions of a plurality of gaze points detected by the gaze position detection unit, determine a gaze area so as to include the plurality of gaze points, and extract a partial image including the gaze area from the captured image, A display captured image generation unit that outputs the extracted partial image as the display captured image;
A display image generation unit that generates the display image by arranging the display captured image in an undrawn area of the driving support information in a display image area corresponding to the display image;
A head-up display device comprising: By using at least the detection result of the line-of-sight detection unit, the occupant's line-of-sight has a virtual image external gaze detection unit that determines whether or not the line of sight of the occupant is directed outside the virtual image of the display image,
The gaze position detection unit detects the position of the occupant's gaze point when the virtual image gaze detection unit determines that the sight line of the occupant is directed outside the virtual image of the display image. The head-up display device according to claim 1. Whether or not the sight line of the occupant is directed outside the virtual image of the display image or whether or not the virtual image of the driving support information is directed by using at least the detection result of the visual line detection unit A virtual image gaze detection unit for determining whether
The display image generation unit
When it is determined that the sight line of the occupant is directed to the virtual image of the display image or the virtual image of the driving support information, the display captured image is enlarged or maintained at the current size. Placed in the undrawn area of the driving support information,
When it is determined that the sight line of the occupant is directed outside the virtual image of the display image or outside the virtual image of the driving support information by the virtual image outside gaze detection unit, the display captured image is reduced and the driving is performed The head-up display device according to claim 1, wherein the head-up display device is arranged in an undrawn area of support information, or the display captured image is not displayed. The display control unit includes a priority order assigning unit that assigns a priority order according to importance for the components of the driving support information and the display captured image.
The display image generation unit reduces a component having a lower priority than the display captured image among the components of the driving support information, and arranges the display image in an undrawn area of the driving support information. The head-up display device according to claim 1. The display control unit includes a priority order assigning unit that assigns a priority order according to the importance for the main constituent elements included in the driving support information and the captured image,
The display captured image generation unit determines a gaze area so as to include the plurality of gaze points based on positions of the plurality of gaze points detected by the gaze position detection unit, and is given by the priority order giving unit Referring to the priority order, a partial image is extracted from the captured image so that at least one of the main constituent elements of the captured image has a higher priority order is included and the gaze area is included. The head-up display device according to claim 1, wherein the extracted partial image is output as the display captured image.   The display image generation unit determines a priority order of the display captured image based on the priority order given by the priority order assigning unit, and includes the display captured image among the components of the driving support information. The head-up display device according to claim 5, wherein those having a priority lower than the priority are reduced, and the displayed captured image is arranged in an undrawn area of the driving support information. A biological information detection unit for detecting the biological information of the occupant;
4. The head-up according to claim 2, wherein the virtual image extraneous gaze detection unit performs the determination based on at least one of a detection result of the visual line detection unit and a detection result of the biological information detection unit. Display device.

Images