WO2016088227A1

WO2016088227A1 - Video display device and method

Info

Publication number: WO2016088227A1
Application number: PCT/JP2014/082025
Authority: WO
Inventors: 大西　邦一
Original assignee: 日立マクセル株式会社
Priority date: 2014-12-03
Filing date: 2014-12-03
Publication date: 2016-06-09

Abstract

The objective of the present invention is to provide a technology that improves the visibility of an additional display even if a user changes the viewpoint. In order to accomplish the objective, the present invention generates video information by imaging the external visual scene of a user (S11), and extracts from the video information a specific object 12 information of which is to be provided to the user (S12). In addition, the direction of the line of sight of the user is detected to generate line-of-sight direction information (S02). Then, a direct visual field area that is a visual field when the user views the external visual scene through a display screen is calculated from the line-of-sight direction information of the user through the semi-transparent display screen which is a display screen to display the video information and which allows the user to directly view the external visual scene through the display screen, and an imaging visual field area that is a visual field when the external visual scene is imaged is calculated (S03). From an additional information storage unit that stores therein, in an associated manner, additional information indicating information on specific objects, the additional information that is associated with the extracted specific object is extracted (S17). A relative positional relationship between the imaging visual field area and the direct visual field area is calculated (S18), and by using the relative positional information, the extracted additional information is displayed on the display screen so as to follow the movement of the direct visual field area (S19).

Description

Video display apparatus and method

The present invention relates to an image display apparatus and method, and more particularly, to a technique that is mounted on a user's head and displays predetermined image information in front of the user's eyes.

In recent years, an information video display system using a so-called head mounted display device (a head mounted display is referred to as an HMD) of a goggles type or glasses type that is worn on a user's head has been rapidly spreading. In particular, a so-called see-through type HMD device having a function of directly displaying an external scene through a translucent display screen while a predetermined information image is superimposed on the directly-viewed scene is directly viewed by the user. Since the predetermined related information can be additionally displayed on the external scene, the application range is wide.

As an example of a technique for performing additional display on an HMD device, Patent Document 1 states that “in an external light transmissive head-mounted display, an imaging device that electronically captures an object in the line-of-sight direction, and the object is recognized from this electronic data. Image recognition means for displaying information relating to the object stored in advance in the storage means, and enabling to register in advance the template data corresponding to the electronic data captured by the imaging device in the imaging device during the video recognition. "(Summary excerpt).

JP 2006-267887 A

The HMD device described in Patent Document 1 performs additional display by identifying the type of a specific object captured by a camera. However, when the user of the HMD device moves the viewpoint, the position of the user's visual field area changes before and after the viewpoint movement, and the positional relationship between the additional display and the target object changes in the user's visual field area, and visibility decreases. There is a problem of doing. This viewpoint movement of the user is not considered at all in Patent Document 1.

The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a technique for improving the visibility of the additional display even when the user moves the viewpoint.

In order to achieve the above object, the present invention captures a user's external scene to generate video information, and extracts from the video information the specific target that is a target for providing information to the user. Then, it is a display screen that detects the user's line-of-sight direction to generate line-of-sight information and displays the video information, and the user can directly view the external scene through the display screen. Photographing the external scene by calculating a direct visual field area, which is a visual field when the user visually recognizes the external scene through the display screen, from the gaze direction information of the user via a transparent display screen The additional field associated with the extracted specific object is extracted from the additional information storage unit that calculates and captures the field of view of the photographic field that is the field of view when the image is captured. The above A configuration in which a relative positional relationship between a shadow visual field region and the direct visual field region is calculated, and the extracted additional information is displayed on the display screen by following the movement of the direct visual field region using the relative positional relationship. It is characterized by that.

According to the present invention, it is possible to provide a technique for improving the visibility of the additional display even when the user moves the viewpoint. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

Schematic diagram showing an example of an HMD system in the first embodiment The perspective view which showed the outline | summary of the external appearance structural example of the HMD apparatus which is the principal part of the HMD system in 1st embodiment, and bears the function of a video display apparatus. The figure which shows the hardware constitutions of the danger avoidance warning apparatus 5 Schematic block diagram showing the electronic circuit configuration of the danger avoidance warning device The flowchart which shows the flow of a process of the driving assistance system which concerns on 1st embodiment. It is a figure for demonstrating a risk determination process, Comprising: (a) is the picked-up image schematic of the electronic camera 3 for outside scene photography which caught the moment when the pedestrian is going to jump out from the side road ahead of the own vehicle. (B) is the schematic diagram which looked down at the condition of Fig.6 (a). Diagram showing the principle of positional deviation between the visual field of the displayed image (field of view of the shooting area) and the direct visual field of view Schematic showing an example of the relationship between the visual field area of the displayed video and the direct visual field area of the driver FIG. 6 is a schematic diagram showing a case where the display visual field direction and the direct visual field direction of the driver are deviated, where (a) shows a state before correction of the deviation of the dynamic visual field direction, and (b) shows a state after correction. It is a figure which shows the additional display example in case a monitoring target object exists in a driver | operator visual recognition visual field area, Comprising: (a) shows the additional display example with respect to a pedestrian's jumping out, (b) is an additional display example with respect to a red signal. is there. Figure showing another example of additional display The schematic block diagram which showed the electronic circuit structure of the driving assistance device which concerns on 2nd embodiment. The flowchart which shows the flow of operation | movement of the driving assistance system which concerns on 2nd embodiment. The figure which shows the example of a screen display of the display screen of the HMD apparatus which concerns on 2nd embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Throughout the drawings, the same components are denoted by the same reference numerals, and redundant description is omitted.

<First embodiment>
The first embodiment is an embodiment in which the present invention is applied to a driving support system that supports a driver who drives a vehicle using an HMD device, and a danger avoidance warning item is additionally displayed. First, a schematic configuration of the present embodiment will be described with reference to FIGS. 1 and 2. Here, FIG. 1 is a schematic diagram showing an example of the HMD system in the first embodiment. FIG. 2 is a perspective view showing an outline of an external configuration example of an HMD device that is a main part of the HMD system according to the first embodiment and has a function of a video display device.

The driving support system 10 shown in FIG. 1 includes the HMD device 1 that is mounted on the head of the driver 20 driving the automobile 50. The HMD device 1 is connected to a danger avoidance warning device 5 as shown in FIG. In FIG. 2, for the convenience of explanation, the danger avoidance warning device 5, which is a separate device from the HMD device 1, is illustrated with a wired connection, but may be wirelessly connected or configured integrally with the HMD device 1. May be.

The HMD device 1 includes a so-called see-through that includes a transflective display screen 2 having a function of displaying an image in the visual field of the driver 20 in a state where the outside world in front of the eyes of the driver 20 is visible. This is a type of HMD device 1. Further, the HMD device 1 holds a display screen 2 and is mounted on a user's head, a mounting body 1a, an electronic camera 3 for photographing an outside scene, and an eyeball camera including an electronic camera that images the eyeball of the driver 20, It has. When the user wears the mounting body 1a on the head, the state in which the display screen 2 is positioned in front of the eyes can be maintained.

The outside scene photographing electronic camera 3 is installed so as to photograph a scene of the outside world from almost the same direction as the line of sight of the driver 20 to which the HMD device 1 is mounted. Has the ability to shoot.

The outside scene photographing electronic camera 3 may be provided with a normal visible light photographing function, or may be provided with a non-visible light photographing function such as infrared rays.

Further, the outside scene photographing electronic camera 3 may be provided with a distance measuring unit for measuring the distance between the subject to be photographed and the outside scene photographing electronic camera 3, such as an autofocus function or an infrared distance measuring sensor.

The eyeball camera is, for example, a small electronic camera arranged at a position that does not obstruct the field of view of the driver 20. The eyeball camera captures the eyeball of the driver 20 obliquely from the front, and the driver or eye movement of the driver 20 in the eyeball image is viewed from the driver. 20 line-of-sight directions are detected. The eyeball image is output to a user gaze direction calculation unit 106 of the danger avoidance warning device 5 described later, and a gaze direction calculation process is executed. The gaze direction detection device is not limited to the configuration including the eyeball camera and the user gaze direction detection unit, and any configuration can be used as long as it can detect the gaze direction of the driver 20 or the visual field area visually recognized by the driver 20. It may be a configuration.

The hardware configuration of the danger avoidance warning device 5 according to the present embodiment will be described with reference to FIG. FIG. 3 is a diagram illustrating a hardware configuration of the danger avoidance warning device 5.

As shown in FIG. 3, the danger avoidance warning device 5 includes a CPU (Central Processing Unit) 51, a RAM (Random Access Memory) 52, a ROM (Read Only Memory) 53, an HDD (Hard Disk Drive) 54, an I / F 55, And a bus 58. The CPU 51, RAM 52, ROM 53, HDD 54, and I / F 55 are connected to each other via a bus 58.

The danger avoidance warning device 5 is connected to each of the display screen 2, the outside scene photographing electronic camera 3, and the eyeball camera via the I / F 55.

Next, the internal configuration of the danger avoidance warning device 5 will be described with reference to FIG. FIG. 4 is a schematic block diagram showing an electronic circuit configuration of the danger avoidance warning device of the danger avoidance warning device. The function of each part of this block diagram will be described below.

The danger avoidance warning device 5 includes a main control unit 100, a specific object extraction unit 101, a specific object related information detection unit 102, an imaging visual field region calculation unit 103, a monitoring object determination unit 104, and a monitoring object risk degree determination unit 105. , User gaze direction calculation unit 106, user visual field calculation unit 107, display video control unit 108, memory 109, graphic memory 110, own vehicle related information acquisition unit 112, map data memory 113, and GPS (Global Positioning System) A device communication antenna 60 is included. Among these configurations, the main control unit 100, the specific object extraction unit 101, the specific object related information detection unit 102, the imaging visual field region calculation unit 103, the monitoring target determination unit 104, the monitoring target risk determination unit 105, The user visual field direction calculation unit 106, the user visual field region calculation unit 107, the display video control unit 108, and the own vehicle related information acquisition unit 112 include software that realizes the functions of these blocks, and the hardware illustrated in FIG. Are configured by collaboration. Each of the memory 109, the graphic memory 110, and the map data memory 113 is configured by the RAM 52 and / or the ROM 53.

The specific object extraction unit 101 is connected to the electronic camera 3 for shooting an outside scene. This specific object extraction unit 101 uses a video image of a specific object such as a moving person, a car, a traffic light, a railroad crossing, a road sign, a lane, or the like from an outside scene image captured by the outside scene photographing electronic camera 3. A function is provided for identifying the type of the specific object extracted by extracting the information and collating it with the video information for specific object collation stored in the memory 109 in advance.

On the other hand, the electronic camera 3 for photographing outside scenes has a specific target for detecting related information such as the moving direction and the moving speed of the specific target in addition to the specific target extracting unit 101 if it is moving. An object-related information detection unit 102 and an imaging field area calculation unit 103 that detects the imaging field of view of the outside scene electronic camera 3 are connected to each other.

The monitoring object determination unit 104 has a risk that the specific object will pose a danger to the traveling of the vehicle in the future based on the information acquired by the specific object extraction unit 101 and the specific object related information detection unit 102. In addition, it has a function of determining whether or not it is a monitoring object that needs to be continuously monitored.

The monitoring object risk degree determination unit 105 includes information related to the monitoring object obtained from each of the above parts, and the vehicle related information such as the vehicle position acquired by the vehicle related information acquisition unit 112 and the traveling speed and direction of the vehicle. Whether the risk level of the monitored object with respect to the vehicle is comprehensively determined from each data such as map data around the vehicle position stored in the map data memory 113, and a danger avoidance warning is displayed to the driver It has a function to decide whether or not.

In this embodiment, the GPS device communication antenna 60 is connected to the own vehicle related information acquisition unit 112. The vehicle 50 is equipped with a GPS device (not shown) that receives a positioning radio wave from a GPS satellite and detects the position of the vehicle, and receives GPS data from the GPS device via the GPS device communication antenna 60. The configuration for acquiring the vehicle-related information is not limited to this, and an inertial measurement device (Inertial Measurement Unit), a vehicle traveling speed detection device mounted on the automobile 50, and the like may be used in combination. Moreover, you may include the GPS apparatus in the driving assistance system 10 which concerns on this embodiment.

The monitoring object risk determination unit 105 determines whether to display a danger avoidance warning. Therefore, the monitoring object risk degree determination unit 105 corresponds to an additional display determination unit. The display video control unit 108 performs control to select and extract a predetermined danger avoidance warning item from the graphic memory 110 and to superimpose it on a predetermined position on the display screen 2 of the HMD device 1.

At this time, the displayed danger avoidance warning item is a display item for a monitoring object within the visual field that the driver 20 himself / herself directly views through the display screen 2 of the HMD device 1 or a monitoring target object outside the driver's visual field. It is preferable that the driver 20 can recognize it correctly. For this purpose, the danger avoidance warning item superimposed on the display screen 2 of the HMD device 1 is correctly positioned in the vicinity of the monitoring object that the driver 20 is directly viewing or in the display position associated with the monitoring object. Need to be displayed.

In order to realize this, in the present embodiment, a user's line-of-sight direction calculation unit 106 that calculates the line-of-sight direction of the user, that is, the driver 20, of the eyeball camera and the HMD device 1 as the line-of-sight detection device, and the driver uses the line-of-sight information And a user visual field area calculation unit 107 that detects a visual field area that is directly viewed through one display screen 2. The relative visual relationship between the direct visual field of view of the driver obtained by the user visual field calculation unit 107 and the photographing visual field of the outside scene photographing electronic camera 3 obtained by the photographing visual field region calculation unit 103 is constantly monitored. The correct display position of the danger avoidance warning item to be displayed in the display image control unit 108 is calculated from the monitor data. A specific example of the danger avoidance warning item display position calculation processing flow will be described in detail later.

Note that the function and operation of each unit in the block are controlled by the main control unit 100.

Next, the operation of the driving support system according to the first embodiment will be described using the flowchart of FIG. FIG. 5 is a flowchart showing a process flow of the driving support system according to the first embodiment. In starting the following processing, it is assumed that the main power supply of the HMD device 1 and the danger avoidance warning device 5 is turned on. When the main power is turned on, the user visual field detection process from steps S01 to S03 and the danger avoidance warning process from steps S11 to S20 are started in parallel. Hereinafter, a user visual field detection process and a danger avoidance warning process will be described in this order.

In the user visual field detection process, the visual line direction detection device 4 captures an image (eyeball image) of the eye of the user, that is, the driver 20 (step S01), and outputs it to the user visual line direction calculation unit 106.

The user gaze direction calculation unit 106 performs, for example, black eye detection processing from the eyeball image, detects the user's gaze direction from the position and direction of the black eye (step S02), and outputs the gaze direction to the user visual field region calculation unit 107. To do.

The user visual field area calculation unit 107 detects the visual field area of the driver 20 based on the line-of-sight direction information (step S03). For example, the intersection of the vector indicating the line-of-sight direction and the display screen 2 is determined as the user's visual field center, and a predetermined in-plane range of the display screen 2 from the visual field center is defined as the user visual field region. Thereafter, the process proceeds to step S19.

On the other hand, in the danger avoidance warning process, the outside scene photographing electronic camera 3 captures an image of the outside scene (hereinafter referred to as “outside scene image”) from substantially the same viewpoint as the driver 20 (step S11), and the specific object extraction unit 101, the specific object The information is output to the object related information detection unit 102 and the photographing visual field region calculation unit 103.

The imaging field area calculation unit 103 detects an area (hereinafter referred to as “imaging field of view”) captured as an outside scene image based on the outside scene image (step S12), and then proceeds to step S18.

The specific object extraction unit 101 extracts video information of a specific object such as a moving person, a car, a traffic light, a railroad crossing, a road sign, a lane, and the like on a road from an outside scene image, and is stored in a memory in advance. The type of the specific object extracted by collating with each specific object collation video information is identified. The specific object extraction unit 101 outputs the extracted specific object to the specific object related information detection unit 102.

Furthermore, the specific object related information detection unit 102 detects the related information of the specific object acquired from the specific object extraction unit 101, for example, the moving direction and the moving speed of the specific object (step S13). The related information detection method may be detected based on, for example, the amount of change in a region where the specific object is captured between successive frames, or an output value from a distance measuring unit that measures the distance to the specific object. The related information may be detected based on the above. The extracted specific object and the related information about it are output to the monitoring object determination unit 104.

The monitoring object determination unit 104 determines whether or not the specific object extracted and identified in step S13 is a monitoring object having a risk of danger with respect to traveling of the host vehicle (step S14). If the determination result is “Yes”, the process proceeds to the next step 16. On the other hand, if the determination result is “No”, the process returns to step S01 and step S11. For example, based on the moving direction and speed of the monitored object, the determination as to whether or not the object is a monitored object is performed when a vector indicating the moving direction of the monitored object intersects with the moving direction vector of the own vehicle (the vector length indicates the moving speed). It is determined that there is a possibility of interference between the own vehicle and the monitoring object.

The monitoring object determination unit 104 continues to acquire related information such as the relative position, relative speed, and relative movement direction of the specific object that has been determined as the monitoring object in step S14 (step S15). Output to the risk determination unit 105.

The monitored object risk level determination unit 105 determines the risk level of the monitored object on the vehicle from the related information of the monitored object obtained in step S13, and determines whether or not to display a predetermined danger avoidance warning item. Determination is made (step S16). If the determination result is “Yes”, the process proceeds to the next step S17. On the other hand, if the determination result is “No”, the process returns to step S01 and step S11. Specific processing contents in steps S15 and S18 will be described later.

The monitored object risk level determination unit 105 determines the type of danger avoidance warning item to be displayed (S17). The monitoring object risk level determination unit 105 may display a danger avoidance warning item that can alert the user more strongly when the distance to the own vehicle is closer even if the monitoring object is the same. .

The display video control unit 108 determines the display position in the display screen 2 based on the user's visual field information and photographing visual field information (step S18), and additionally displays a warning item in the display screen 2 (step S19). . If the main power supply of the driving support system 10 is ON, the processing of the driving support system 10 is continued (step S20 / Yes), and the process returns to steps S01 and S11. The process of the driving support system 10 in which the main power is turned off is terminated (Step S20 / No).

Next, specific processing contents in step S15 and step S16 of the processing flow will be described by way of an example in which a pedestrian jumps out of a side road ahead of the vehicle. 6A and 6B are diagrams for explaining the risk determination process. FIG. 6A is an outline of a captured image of the electronic camera 3 for capturing an outside scene that captures a moment when a pedestrian is about to jump out of the side of the road. It is a figure and (b) is the schematic diagram which looked down on the condition of Fig.6 (a).

As shown in FIG. 6A, the specific object extraction unit 101 extracts and identifies an image of a pedestrian 31a that is about to jump out from the outside scene image captured in the imaging field of view area 11 in the forward direction of the host vehicle. The specific object related information detection unit 102 detects related information related to the pedestrian 31a. The monitoring object determination unit 104 determines that this is a monitoring object and enters a mode for monitoring its movement. Then, in the subsequent shooting frames, when the pedestrian 31a approaches the front direction of the host vehicle, for example, as shown in FIG. A pedestrian movement vector “u” is calculated from

points

41a and 41b corresponding to the position coordinates of the

pedestrians

31a and 31b on the map data 40 around the own vehicle stored in the data memory 113, and A movement vector “v” of the point 42 corresponding to the own vehicle is calculated on the map data 40 from the vehicle related information. By comparing the direction and magnitude (corresponding to speed) of the vector “u” and the vector “v”, the collision risk between the target pedestrian and the subject vehicle is calculated, and the danger avoidance is performed according to the collision risk. It is determined whether or not the warning item is displayed in the HMD screen.

Next, the display position of the danger avoidance warning item will be described. Each danger avoidance warning item superimposed and displayed on the display screen 2 of the HMD device 1 is directly visible through the display screen 2 by the driver 20, or is near or near the corresponding monitoring object outside the field of view. From the viewpoint of improving the visibility, it is preferable that the information is accurately superimposed and displayed at a predetermined position associated with the object as much as possible.

As described above, in order to accurately display a predetermined danger avoidance warning item at a position corresponding to the monitoring target in the scene that the driver 20 is directly viewing through the display screen 2, the display is superimposed on the display screen 2. It is necessary that the viewing direction of the image to be displayed and the viewing direction of the outside scene that is directly viewed by the driver match completely.

However, the visual field area directly seen by the driver 20 wearing the HMD device 1 is always changing according to the line-of-sight direction in which the driver 20 is gazing. On the other hand, the outside scene photographing electronic camera 3 fixed to the HMD device 1 is fixed in the direction in which the HMD device 1 is facing, that is, the direction in which the head of the driver 20 is facing. For this reason, there is a possibility that the visual field direction of the photographing visual field region formed from the photographed video of the outside scene photographing electronic camera 3 and the direct visual field of view of the driver 20 itself are easily shifted. More specifically, the field of view for photographing and the field of view of the display image captured with the field of view differ depending on the position on the display screen 2 where the image is displayed.

Here, the difference between the field of view of the photographing field and the field of view of the display image is that the center point of the photographing field of view and the center point of the display region of the display screen 2 are fixed because the electronic camera 3 for photographing the outside scene is fixed to the HMD device 1. The correction amount of the error can be found from the geometric positional relationship. Therefore, the center point of the photographing visual field area and the center point of the display screen 2 are treated as being coincident using the correction amount. Accordingly, in the following description, it is assumed that the visual field of the display image and the photographing region visual field are the same. Accordingly, the principle of occurrence of positional deviation between the visual field of the display image (imaging area visual field) and the direct visual field of vision will be described with reference to FIG. FIG. 7 is a diagram showing the principle of occurrence of positional deviation between the visual field of the display image (imaging area visual field) and the direct visual field of view.

The upper part of FIG. 7 shows a state where the driver 20 is facing the front. The fact that the driver 20 visually recognizes an object to be monitored, for example, a pedestrian 31, means that visible light reflected by the surface of the pedestrian 31 is incident on each of the left eye 21L and the right eye 21R, and in the left eye 21L and the right eye 21R. It is in a state where it reaches within each retina and forms an image. The path of incident light is indicated by the symbol β. The middle point of the intersection of each path β of visible light incident on the left and right eyes of the pedestrian 31 and the display screen 2 is indicated by Q.

The lower part of FIG. 7 shows a state where the driver 20 is facing right. Q ′ represents the midpoint of the intersection of the path β ′ of the incident light and the display screen 2 when the visible light reflected by the surface of the pedestrian 31 enters each of the left eye 21L and the right eye 21R. The midpoint Q ′ is different in position from the midpoint Q. That is, even if the relative positions of the HMD device 1 and the pedestrian 31 are the same, if the line of sight of the driver 20 deviates from the vector α ′ (upper part in FIG. 7) to the vector α ′ (lower part in FIG. 7), the display screen 2 The visual field of the display image of the pedestrian 31 and the direct visual field of view of the driver 20 are shifted.

Therefore, in this embodiment, the midpoint P of the intersection point between the line-of-sight vector α of each of the left and right eyes of the driver 20 and the display screen 2 is determined as the center of the display field. In the upper part of FIG. 7, the point P is the center of the display field, and in the lower part of FIG. 7, the point P ′ is the center of the display field. Therefore, the point P is at a position that is displaced by Δx in the horizontal direction from the point P on the surface of the display screen 2. In FIG. 7, the horizontal cross section is described as an example, but the line of sight also changes in the vertical direction. In this case, the point P is at a position that is displaced by Δy in the vertical direction from the point P on the surface of the display screen 2.

Thus, even if there is no change in the relative position between the HMD device 1 and the monitoring target due to the viewpoint movement of the driver 20, there is a deviation between the visual field direction of the display image and the direct visual field direction of the driver (user visual field direction). Arise. Therefore, in order to always match these, it is necessary to perform some dynamic visual field direction correction for adjusting the visual field direction of the displayed image to the direct visual field direction of the driver in real time.

Therefore, in the present embodiment, the display visual field direction dynamic correction process is performed as follows. First, the field of view of the electronic camera 3 for photographing the outside scene is set to be sufficiently wider than the field of view directly visible through the display screen 2 by the driver 20. Thereafter, dynamic correction of the display visual field direction as described with reference to FIGS. 8 and 9 is performed.

FIG. 8 is a schematic diagram showing an example of the relationship between the display field of view of the HMD device 1 and the direct visual field of view of the driver 20. A region surrounded by a broken-line square frame in FIG. 8 is the entire region of the outside scene video, and coincides with the photographing field region 11 of the outside scene photographing electronic camera 3. An area surrounded by a solid square frame is a display visual field area 12 of the HMD device 1, and an area surrounded by an elliptical frame is a direct visual field area 13 of the driver 20. In the center of the display visual field area 12, a point O and a virtual XY coordinate axis with this point O as the origin are set. The display position (coordinates) of each display item is calculated with reference to this XY coordinate axis. The origin O and the XY coordinate axes are virtually set in the processing step S18 in the flowchart of FIG. 4 and are not actually displayed on the display screen 2.

On the other hand, in the same processing step S18, the center point of the visual field region is determined from the direct visual field region 13 of the HMD user, that is, the driver 20, determined through the eye camera, the user's visual line direction calculation unit 106, and the user visual field region calculation unit 107. P is obtained, and the coordinate position on the XY coordinates provided in the display visual field region 12 is calculated.

For example, in the example of FIG. 8, the coordinates of the point P are (0, 0), that is, the center point P of the direct visual field region 13 of the driver 20 coincides with the origin O of the display visual field region 12. In this case, obviously, the display visual field direction of the HMD device 1 and the direct visual field direction of the driver coincide with each other.

On the other hand, FIG. 9 is a schematic diagram showing a case where the display visual field direction of the HMD device 1 is shifted from the direct visual field direction of the driver. In this case, as shown in FIG. The coordinates of the center point P of the area 13 are coordinates (Xp, Yp) deviating from the origin O of the display visual field area 12.

At this time, as shown in FIG. 9B, the system automatically shifts the origin O of the display visual field region 12 to the same position O ′ as the position of the point P, and a new coordinate axis X having this O ′ as the origin. Define '-Y'. In other words, the display image control unit 108 adjusts the installation position of the origin O of the display visual field region 12 and the coordinate axes XY so that the center point P of the driver visual recognition visual field region 13 always coincides with the origin of the display visual field region 12. The display position (coordinates) of each display item is calculated using the new XY coordinate axes after the adjustment.

By automatically adjusting the origin of the display field and the display coordinate axis in accordance with the driver's direct field of view, each display item can be accurately displayed at a position corresponding to the target object to be directly viewed by the driver. .

Referring to FIGS. 10 and 11, a display example of the danger avoidance warning item in the driving support system according to the first embodiment will be described. FIG. 10 is a diagram showing an example of additional display when a monitoring target is present in the driver visual field of view, wherein (a) shows an example of additional display for a pedestrian jumping out, and (b) is an addition to a red signal. It is a display example.

FIG. 10A is an example in which the pedestrian described in FIG. 6 is about to jump out from the side road in the forward direction of the host vehicle. In order to concentrate the driver's attention on the pedestrian that is the monitoring target, A danger avoidance warning item such as a frame line 201 displayed so as to surround a pedestrian (displayed with a broken line) 201 and a warning sentence 202 of “jump out caution!” Displayed near the center of the field of view is superimposed.

FIG. 10B is an example in which the traffic light in front of the host vehicle is a red signal. For example, in order to concentrate the driver's attention on the red signal that is the object to be monitored, the signal is displayed so as to surround the red signal in the front. Danger avoidance warning items such as a framed line (displayed by a broken line) 203 and a warning sentence 204 of “Signal light red light stop!” Displayed near the center of the visual field are superimposed.

10 (a) and 10 (b) show examples of danger avoidance warning items for danger monitoring objects in the direct visual field region 13 of the driver 20, but in the present invention, an electronic camera for photographing outside scenes By taking the 3 field of view sufficiently wider than the driver's direct field of view 13, it is possible to display a danger avoidance warning item even for a monitoring object outside the driver's visual field of view, and alert the driver. It is also possible.

FIG. 11 is a schematic view showing such a display example. The pedestrian is about to jump out from the left outside of the direct visual field area 13 of the driver 20 in the forward direction of the vehicle. Naturally, the driver 20 cannot visually recognize the pedestrian, but the danger avoidance warning device 5 of the present invention quickly identifies the pedestrian as a monitoring object from the captured image of the outside scene photographing electronic camera 3 and collides from the movement. It is determined that there is a danger risk, and a danger avoidance warning item such as an arrow 205 for alerting or a warning sentence 206 of “Jump out from the left!” Is displayed near the center in the direct visual field area 13 of the driver.

The example of FIG. 11 is an example of a pedestrian jumping out, but is not limited to this, and a red signal as in the example of FIG. 10B is outside the direct visual field region 13 of the driver 20. Even in some cases, similar danger avoidance warning items can be displayed.

In addition, monitoring objects are not limited to pedestrians jumping out and traffic lights, but targets that may cause some danger risk in vehicle driving such as car jumping out, road signs, railroad crossings, road steps, lanes, etc. Any object can be used as long as it is an object. A more advanced vehicle driving support system can be realized by selecting and displaying appropriate danger avoidance warning items from information such as the type, position, and movement of each monitored object.

In this way, by displaying the danger avoidance warning item superimposed in the field of view directly visible to the driver, the responsiveness of danger risk avoidance is significantly improved compared to the driving support system using the conventional vehicle built-in display. Furthermore, by incorporating such a system into the HMD device, the driver can always drive in an appropriate danger avoidance warning environment even when driving a vehicle without a driving support system.

In addition, according to the present embodiment, not only the type of the monitoring object but also related information such as the position, moving speed, and moving direction is acquired, and the risk avoidance warning item taking this into consideration is displayed, thereby promptly responding to the driver. The convenience of the driving support system can be further improved.

Further, in the present embodiment, when performing so-called additional display in which the danger avoidance warning item is directly superimposed on the visual field, the viewpoint movement of the driver is detected, and the origin of the direct visual field area and the display visual field area associated with the movement of the visual point are detected. Corrects misalignment and performs additional display. Thereby, since there is no position shift between the monitoring object and the danger availability warning item as seen from the driver, or it is mitigated, the recognizability of the driver can be improved.

<Second embodiment>
The second embodiment is an embodiment in which the present invention is applied to a driving route guidance system, that is, a so-called navigation system, among driving assistance systems. The second embodiment will be described below with reference to FIGS. FIG. 12 is a schematic block diagram illustrating an electronic circuit configuration of the driving support apparatus according to the second embodiment. FIG. 13 is a flowchart showing a flow of operations of the driving support system according to the second embodiment. FIG. 14 is a screen display example of the display screen of the HMD device according to the second embodiment. The function of each part of this block diagram will be described below. In this figure, blocks having the same functions as those in the schematic block diagram of the first embodiment shown in FIG.

As shown in FIG. 12, the difference between the driving support device 5 a according to the second embodiment and the danger avoidance warning device 5 according to the first embodiment is that the degree of risk of the monitoring object included in the danger avoidance warning device 5. The determination unit 105 is not provided, and a route determination unit 111 is provided instead. Further, even though the block names are the same, there are differences in the objects recognized as the specific objects, which will be described in detail below.

As shown in FIG. 12, the specific object extraction unit 101 is connected to the electronic camera 3 for photographing outside scenes. This specific object extraction unit 101 is often used along a road such as a road sign, a traffic light, an intersection, a railroad crossing, or a convenience store, a family restaurant, a gas station, etc. Video information that becomes a search point when searching for a travel route such as a signboard of a store to be seen is extracted, and the specific object extracted by collating with the video information of each specific object stored in the memory 109 in advance A function for identifying the type is provided.

On the other hand, in the outside scene photographing electronic camera 3, in addition to the particular object extracting unit 101, the specific object related information detection unit 102 that detects related information such as the position of the specific object and the outside scene photographing electronic camera 3 are photographed. A photographing visual field region calculation unit 103 for detecting the visual field is connected.

Next, the monitoring target determination unit 104 needs the specific target video for future route search of the own vehicle from various information acquired by the specific target extraction unit 101 and the specific target related information detection unit 102. It has a function of determining whether or not it is a monitoring target that needs to monitor the relative positional relationship between the specific target and the vehicle.

On the other hand, the route determination unit 111 detects the vehicle-related information acquired by the vehicle-related information acquisition unit 112 that detects the information obtained from the above-described units, the vehicle position, the traveling speed of the vehicle, the traveling direction, and the like. The map data stored in the map data memory 113 is compared with the map data in the vicinity of the own vehicle position to search for the driving route of the own vehicle, and further to guide the driving route to the driver, the driving route for the monitored object It has a function of determining whether to display guidance items.

In the present embodiment, the vehicle-related information acquisition unit 112 acquires, for example, GPS data. However, the present invention is not limited to this example. For example, the vehicle-related information acquisition unit 112 is not limited to this example. You may provide the structure which detects an own vehicle position, an own vehicle travel speed, a direction, etc. in an HMD system. The own vehicle related information acquisition unit 112 corresponds to a position information acquisition unit that acquires position information indicating the current position of the user.

When the display of the predetermined travel route guidance item is determined by the route determination unit 111, the display image control unit 108 selects and extracts a predetermined travel route guidance item such as an arrow indicating a left turn or a right turn from the graphic memory 110, and extracts it. Control is performed so as to superimpose and display on the display screen 2 of the HMD device 1.

At this time, as in the first embodiment, the displayed travel route guidance item can be correctly recognized as a display item associated with a corresponding monitoring object that the driver 20 himself / herself directly views through the display screen 2. It is necessary to accurately superimpose and display at a predetermined position in the display screen 2.

In order to realize this, in the present embodiment, the driver directly recognizes the user's line-of-sight direction calculation unit 106 that detects the line-of-sight direction of the user of the HMD device 1, that is, the driver through the eyeball camera and the line-of-sight information directly through the display screen 2. A user visual field calculation unit 107 that detects the visual field to be detected is provided. Then, the relative position relationship between the direct visual field of view of the driver obtained by the user visual field calculation unit 107 and the photographing visual field of the outside scene photographing electronic camera 3 obtained by the photographing visual field region calculation unit 103 is constantly monitored. From the data, the correct display position of the danger avoidance warning item to be displayed in the display image control unit 108 is calculated. Note that a specific example of the above warning item display position calculation processing flow has already been described in the first embodiment (FIGS. 8 and 9), and thus the description thereof is omitted here.

As in the first embodiment, the function and operation of each unit in the block are controlled by the main control unit 100.

Next, the processing flow of the so-called navigation system according to the second embodiment will be described with reference to the flowchart of FIG. In this figure, the same numbers are assigned to the same processing steps as the processing steps in the flowchart of FIG. 5 shown in the first embodiment.

In the user visual field detection process, the visual line direction detection device 4 captures an image of the eye of the user, that is, the driver 20 (user visual line image) (step S01), and outputs it to the user visual line direction calculation unit 106. Then, the user gaze direction calculation unit 106 detects the gaze direction (step S02), and the user visual field region calculation unit 107 determines the direct visual field region of the driver 20 based on the gaze direction information (step S03).

In parallel with the user visual field detection process, the outside scene photographing electronic camera 3 sequentially acquires predetermined video information (step S11), and the photographing visual field region calculation unit 103 detects the photographing visual field based on the video information (step S12). )

At the same time, the specific object extraction unit 101 uses the above-described route search specific objects from the video information acquired in step S11, such as road signs, traffic lights, intersections, railroad crossings, convenience stores, family restaurants, gas stations, and the like. The video information that becomes the point on the travel route search such as a signboard of a shop that is often seen in the store is extracted, and the type of the specific object extracted by comparing with the video information of each specific object stored in the memory in advance Identify (step S13).

Next, the route determination unit 111 acquires the position on the map data of the specific object extracted and identified in step S13, and acquires the position on the map data of the host vehicle and route data to be traveled in the future (step S21). ).

The route determination unit 111 collates the position of the specific object acquired in step S13 with the position of the own vehicle acquired in step S22 and the route data that the own vehicle should travel (step S22). It is determined whether or not any travel route guidance item, that is, a navigation item is displayed on the object (step S23). And if a determination result is "Yes", it will transfer to the following step S25. On the other hand, if the determination result is “No”, the process returns to step S01 and step S11.

The route determination unit 111 determines the type of navigation item to be displayed (step S24).

The display video control unit 108 displays the display position (origin) on the HMD display screen based on the direct visual field information of the driver 20 detected in step S03 and the photographing visual field information of the outside scene photographing electronic camera 3 detected in step S12. To decide. Then, the display video control unit 108 displays the navigation item at a predetermined position on the HMD display screen determined in step S25 (step S25) (step S26).

If the main power supply of the driving support system 10 is ON, the processing of the driving support system 10 is continued (step S27 / Yes), and the process returns to steps S01 and S11. The process of the driving support system 10 in which the main power is turned off is terminated (Step S27 / No).

FIG. 14 shows a display example of navigation items by the navigation system. In this drawing, reference numeral 11 denotes a photographing field of view of the outside scene photographing

electronic camera

3, and 13 denotes a direct field region 13 that the driver directly views through the HMD display screen.

In FIG. 14, the system first extracts and identifies, for example, an image of a signboard of a specific restaurant (portion surrounded by a broken line in the drawing) from the image of the shooting field of view 11. Then, it is determined from the map data, route data, and own vehicle position data stored therein that it is necessary to turn left at the intersection in front of this restaurant as a route to be run, for example. Therefore, the position of the intersection in front of the restaurant is extracted and identified from the photographed video, and an appropriate comment such as an arrow 207 for prompting a left turn to the intersection position within the driver's visual field of view or a guidance comment such as “turn left at this corner” is appropriate. Navigation items are superimposed.

By overlaying navigation items in the field of view directly visible to the driver in this way, the driver frequently looks at the outside scene and the navigation screen while driving as in a conventional navigation system with a built-in vehicle display. This eliminates the hassle of moving the vehicle and significantly increases the driver's awareness of driving safety and navigation information. Furthermore, by incorporating such a system into the HMD device, the driver can drive under an appropriate navigation environment even when driving a vehicle without a navigation system.

As mentioned above, although the example of embodiment of this invention was demonstrated, the structure which implement | achieves the technique of this invention is not restricted to the said embodiment, Various modifications can be considered. For example, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. These all belong to the category of the present invention. In addition, numerical values, messages, and the like appearing in sentences and drawings are merely examples, and the use of different ones does not impair the effects of the present invention. For example, the danger avoidance warning device of the first embodiment and the navigation system of the second embodiment have been described as independent embodiments in the driving support system by the HMD system of the present invention. It may be a comprehensive driving support system in which two systems are integrated.

The functions and the like of the present invention described above may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Further, the microprocessor unit or the like may be realized by software by interpreting and executing an operation program that realizes each function or the like. Hardware and software may be used together.

Also, the control lines and information lines shown in the figure are those that are considered necessary for the explanation, and not all control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

Moreover, although the first embodiment and the second embodiment described above have been described as an example of a driving support system for a driver who drives an automobile, the present invention is not limited to this. For example, the present invention may be used as a driving support system for a driver who drives a two-wheeled vehicle such as a motorcycle or a bicycle, or the present invention may be used as a general pedestrian safety assistance or destination guidance system. .

The present invention is not limited to the field of driving support, and can be applied to technologies that perform extended display (Augmented Reality, AR). For example, AR technology such as) can be applied regardless of its use. For example, it can be applied to games and pedestrian navigation systems. Specifically, it is also used for the attraction explanation device in the theme park (target age, height, waiting time display, etc.), the explanation device for the exhibits in the museum (however, the shooting function is locked), etc. The invention can be applied.

Furthermore, it can also be used as a safety assistance system when moving visually impaired and visually impaired. In this case, however, the necessary danger avoidance warning and the travel route guidance need to use audio or the like instead of displaying the video item on the HMD display screen. Moreover, it may replace with HMD apparatus and the video display apparatus for extended displays provided with the display screen which has transparency may be sufficient.

DESCRIPTION OF SYMBOLS 1 ... HMD apparatus 2 ... Translucent display screen 3 ... Electronic camera 4 for external scene photography ... Eyeball camera 5 ... Danger avoidance warning device 5a ... Driving assistance device 10 ... Driving assistance system

Claims

An external scene photographing unit for photographing a user's external scene and generating video information;
An additional information storage unit that associates and stores additional information indicating information on the specific object to be provided to the user and information on the specific object;
A specific object extraction unit that extracts the specific object from the video information;
A translucent display screen that displays the video information and allows the user to directly view the external scene through the display screen;
A gaze direction detection unit that detects gaze direction information by detecting the gaze direction of the user;
From the user's line-of-sight direction information, a user visual field region calculation unit that calculates a direct visual field region that is a visual field when the user visually recognizes the external scene through the display screen;
A shooting field area calculation unit that calculates a shooting field area that is a field of view when the external scene shooting unit has shot the external scene;
A display image control unit that extracts the additional information associated with the specific object from the additional information storage unit and displays the additional information on the display screen;
The display image control unit calculates a relative positional relationship between the photographing visual field region and the direct visual field region, and uses the relative positional relationship to cause the additional information to follow the movement of the direct visual field region on the display screen. indicate,
A video display device characterized by that.
The display video control unit obtains a positional deviation amount of the center point of the photographing field area and the center point of the direct field area in the display screen, and the identification when the center point of the photographing field area is used as a reference The display position of the object is corrected using the amount of displacement, the corrected position of the specific object is calculated, and the display position of the additional information is calculated based on the corrected position. ,
The video display device according to claim 1.
Specific object related information detection for generating related information including determination elements for determining whether or not to display the specific object extracted by the specific object extraction unit on the display screen based on the video information. And
A monitoring object determining unit that determines whether or not the specific object is a monitoring object in order to determine whether or not the specific object needs to be displayed based on the related information;
The display video control unit displays additional information related to the monitored object;
The video display device according to claim 1.
The external scene photographing unit generates the video information by capturing the external scene with the photographing visual field region wider than the direct visual field region,
The display video control unit displays additional information for the monitoring target in an area that is included in the photographing visual field area and is not included in the direct visual field area.
The video display apparatus according to claim 3.
A mounting body to be mounted on the user's head, wherein the display screen is positioned in front of the user's eyes in a state in which the mounting body is mounted on the user's head;
Based on the related information of the monitoring target, an additional display determination unit that determines whether or not to display additional information related to the monitoring target;
A position information acquisition unit that acquires position information indicating the current position of the user;
The specific object is a pedestrian, a two-wheeled vehicle, an oncoming vehicle, an immediately preceding vehicle that is running or stopped immediately before the user's own vehicle, a traffic light, an approach from another lane that joins the traveling lane of the own vehicle. At least one of the vehicles,
The specific object related information detection unit generates the related information including at least one of a position, a moving direction, and a moving speed of the specific object,
The monitoring object determination unit determines the specific object as the monitoring object when the predicted movement position of the specific object interferes with the traveling direction of the user based on the related information regarding the specific object. And
The additional display determination unit displays the additional information according to the risk level based on the risk level of interference between the user and the monitoring target based on the user position information and the related information. It is determined that
The video display apparatus according to claim 4.
A mounting body to be mounted on the user's head, wherein the display screen is positioned in front of the user's eyes in a state in which the mounting body is mounted on the user's head;
A position information acquisition unit for acquiring position information indicating the current position of the user;
A route determination unit that obtains map data indicating a map around the current position of the user and predicts the travel route of the user based on the map data; and
The specific object is at least one of a building, a facility, and a road serving as a search point for the travel route,
The specific object related information detection unit generates the related information including a relative position between the specific object and the user,
The route determination unit indicates the predicted travel route when the user approaches the specific object based on the user's position information and the relative position between the specific object and the user. Decide to display additional information,
The video display apparatus according to claim 4.
The line-of-sight direction detection unit detects the position of the user's pupil or glow from the electronic camera that images the user's eyeball and outputs an eyeball image, and detects the user's line-of-sight direction. Including a gaze direction calculation unit to detect,
The video display device according to claim 1.
The external scene photographing unit is an electronic camera that captures the outside scene with visible light or infrared light and outputs video information, and a distance measuring unit that measures a distance between a subject of the outside scene and the electronic camera. including,
The video display device according to claim 1.
Photographing a user's external scene and generating video information;
Extracting from the video information the specific object to be provided with information to the user;
Detecting a gaze direction of the user to generate gaze direction information;
The display screen for displaying the video information, and the user can view the external scene through the translucent display screen through which the user can directly view the external scene. Calculating a direct visual field region, which is a visual field when visually recognizing over, from the gaze direction information of the user;
Calculating a shooting field of view that is a field of view when the external scene is shot;
The additional information associated with the extracted specific object is extracted from the additional information storage unit that associates and stores additional information indicating the information related to the specific object, and the relative relationship between the imaging visual field area and the direct visual field area is extracted. Calculating a positional relationship and displaying the extracted additional information on the display screen following the movement of the direct visual field region using the relative positional relationship;
A video display method comprising: