JP2011197736A - Vision field support device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a vision field support device wherein it is unnecessary to turn a head to a direction of a circumferential image, when a user views the circumferential image, other than the vision field range by a head-mounted display device.SOLUTION: The vision field support device includes a plurality of imaging devices 1-3 for photographing images around a moving body; an integrated image generation device 10 constituting an entire circumferential image from the images captured by the plurality of imaging devices; a display device 11 capable of being mounted onto the head of the user 20; and a head movement detection device 12 for detecting the movement of the head of the user 20; and thereby the user can view the circumferential image, other than the vision field, while scrolling the circumferential image.

Description

  The present invention relates to a head-mounted type visual field support device that displays an all-around image generated from images captured by a plurality of imaging devices attached to a moving body based on user's line-of-sight information.

  Known head-mounted image display devices (head-mounted displays) are those used for rescue operations during disasters, and those used by mobile operators. (For example, see Patent Documents 1, 2, 3, and 4)

JP 2007-286269 A Japanese Utility Model Publication No. 5-75892 JP-A-2-274249 JP-A-7-13084

  A conventional head-mounted display device is premised on acquiring and displaying an image in a direction in which the user is facing, and the field of view is limited to a narrow range. For this reason, when the user sees a surrounding image outside the field of view (dead angle), it is necessary to turn the head in that direction, and the head is moved while the head-mounted image display device is worn. There was a problem that it took a burden.

  Further, in order to simultaneously display an entity (an outside scene) in which the user is directly looking within the field of view and various kinds of information, the image is superimposed on a transmissive display device. For this reason, there was a problem that it was difficult to see the scenery of the outside world.

  The present invention has been made in order to solve such a problem, and provides a visual field support device that reduces the burden on the wearer when viewing a surrounding image outside the visual field range by wearing a head-mounted display device. The purpose is to obtain.

  The visual field support device according to the present invention includes a plurality of imaging devices that capture images around a moving body, an integrated image generation device that synthesizes all-around images from images captured by the plurality of imaging devices, and a user's head A transparent area where the scene can be seen at the center of the field of view seen from the user, and a non-transparent area surrounding the transparent area for displaying the all-around image generated by the integrated image generating device And a head movement detection device that detects movement of the user's head, wherein the display device detects movement information of the user's head detected by the head movement detection device. Based on the above, a process of scrolling the entire surrounding image displayed in the non-transparent area is performed.

  According to the present invention, a display in which a user wears on the head an entity in which what is within the field of view is directly visible, and an image that is outside the user's field of view of the entire surrounding image generated by the integrated image generation. Since it can be seen simultaneously by the device, it can be seen outside the field of view without moving the head, and the burden on the user's body, particularly the neck, can be reduced.

It is a block diagram which shows the structure of the visual field assistance apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the concept of the visual field of the display apparatus of FIG. It is a figure which shows the concept of the visual field seen from the user of the display apparatus of FIG. It is a figure which shows the motion of the display image which concerns on Embodiment 2 of this invention. It is a figure which shows the motion of the display image which concerns on Embodiment 3 of this invention. It is a block diagram which shows the structure of the visual field assistance apparatus which concerns on Embodiment 4 of this invention.

Embodiment 1 FIG.
Hereinafter, an embodiment of the present invention will be described.
FIG. 1 is a block diagram showing the configuration of a field-of-view support apparatus according to Embodiment 1 of the present invention. FIG. 2 is a view showing the concept of the field of view by the display device 11 of the field of view assistance device. FIG. 3 is a diagram showing the concept of the field of view as viewed from the user 20 of the display device 11 in the field of view assistance device.

  In FIG. 1, the visual field support device includes a plurality of imaging devices 1, 2, 3, an integrated image generation device 10, a display device 11, and a head motion detection device 12. The plurality of imaging devices 1, 2, 3, the integrated image generation device 10, the display device 11, and the head motion detection device 12 are provided in the head wearing tool 21, and the head mounted image display device (head) -Mount display). The head mounting tool 21 includes a helmet, a headband, a hat, and the like, and is mounted on the head of the user 20.

  The imaging devices 1 to 3 each capture an image around the moving body. The directions of the line-of-sight axes of the imaging devices 1 to 3 are set so as to face the front right side, the front left side, and the rear, respectively, at intervals of 120 degrees around the head circumference of 360 degrees. The imaging devices 1 to 3 are attached to the front right side, the front left side, and the rear of the head-mounted device 21, for example, outside the moving body on which the user 20 is boarded, and at 120 ° intervals around the head circumference 360 °. Installed. In addition, the imaging devices 1 to 3 are installed directly on the head wearing tool 21 and attached to the front right side, the front left side, and the rear side of the head wearing tool 21 and provided at intervals of 120 degrees around the head circumference 360 degrees. May be. In addition, the angle difference of the visual axis between the adjacent imaging devices 1 to 3 may be an angle other than 120 degrees, and the installation angle may be another angular interval.

  The integrated image generation device 10 is mounted on the head-mounted device 21 and synthesizes an all-around image from images captured by the plurality of imaging devices 1 to 3. The perimeter image is configured by pasting an image within a predetermined viewing angle centered on the line of sight of each imaging device to the entire circumference of the user 20 (the entire circumference range of the surrounding spherical surface). You may have a partial overlap in the image between apparatuses 1-3. The display device 11 is attached in front of the head-mounted device 21 and is mounted at a position facing the line of sight of the user 20. The head movement detection device 12 is mounted on the head mounting tool 21 and detects the movement of the head of the user 20. The head motion detection device 12 is provided with, for example, a gyroscope and an acceleration detector, and detects the head motion of the user 20 by detecting the angular velocity of the head and the direction of the reference axis (gaze axis). The integrated image generation apparatus 10 does not necessarily have to be mounted on the head mounting tool 21.

  FIG. 1 shows an example in which there are three imaging devices. However, this does not limit the number of imaging devices, and two or more imaging devices are sufficient. In FIG. 2, the display device 11 includes a transmissive (see-through) region 30 provided in the center of the field of view (in front of the eye) viewed from the user 20 and a non-transmissive region 31 surrounding the transmissive region 30. ing. The transmissive region 30 is configured by a sun visor provided with, for example, a visible light transmissive resin member. The non-transmissive region 31 is configured by a display screen such as an EL element image display device or a liquid crystal image display device. The display device 11 is provided with a scroll processing unit that scrolls the entire surrounding image displayed in the non-transparent region 31 based on the movement information of the user's head detected by the head motion detection device 12.

The visual field support device is configured as described above and operates as follows.
As shown in FIG. 1, in this visual field support device, an all-around image is generated by an integrated image generation device 10 from images captured by a plurality of imaging devices 1 to 3 attached to a moving body. Based on the movement information of the head of the user 20 detected by the head motion detection device 12, the image in the line-of-sight direction of the user 20 among the all-around images generated by the integrated image generation device 10 is scrolled and scrolled. The image is supplied to the user 20 through the display device 11 attached to the head.

  The visual field of the display device 11 in this visual field support device is as shown in FIG. In the central part of the field of view, a transmission region 30 is arranged for the user 20 to directly see what is within the field of view. Further, around the transmissive area 30, the transmissive area 30 for displaying an image outside the range of the field of view of the user 20 (dead angle) among all the surrounding images generated by the integrated image generation device 10 is surrounded. A non-transmissive region 31 is disposed. It is assumed that the range of the non-transmissive region 31 substantially matches the range of the human field of view.

  FIG. 3A shows a state in which an image around the moving body can be seen through the display device 11 and the window 40 of the moving body. FIG. 3B is a diagram illustrating the movement of the surrounding image of the transmissive region 30 when the transmissive region 30 of the display device 11 is in contact with the end surface 41 of the window 40 of the moving body.

  FIG. 3 shows an example in which the user 20 is on a moving body provided with a window 40. The field of view of the display device 11 viewed from the user 20 is limited by the window end surface 41 of the moving object window 40, and the region outside the window end surface 41 cannot be directly seen by the user 20 visually. . Further, in FIG. 3, surrounding images 50 to 52 of the transmissive region 30 among the all surrounding images generated by the display device 11 are shown. Further, it is assumed that the scenery 53 of the outside world can be seen by the user 20 through the transmission area 30.

  As shown in FIG. 3, when the user 20 moves his / her head to look outside the transmission area 30 of the display device 11, the head motion detection device 12 detects the movement and moves the direction, angular velocity, and the like. Information is output to the display device 11. Based on this movement information, the display device 11 recognizes that the transmission region 30 located at the center of the field of view is in contact with or exceeds the end face 41 of the window of the moving object.

  Specifically, as the movement information, the head is rotating in the right direction of the figure or stopped after rotating, and the direction of the line of sight touches the end face 41 of the window or faces a position beyond it. Recognize that In response to the recognition, the scroll processing unit of the display device 11 includes an external scene 53 that the user 20 can see through the transmissive area 30 and the surrounding images 50 to 52 of the transmissive area 30 displayed in the non-transmissive area 31. Is moved (scrolled) according to the movement of the head.

  In addition, the scroll processing unit of the display device 11 starts moving from the head movement detecting device 12 and moving information such as the direction of the head of the user 20 and the angular velocity, and then the head is rotated in the forward direction and scrolling is started. When reversing in the reverse direction is recognized, processing for stopping scrolling of the surrounding images 50 to 52 of the transmissive area 30 displayed in the non-transmissive area 31 is performed.

  The surrounding images 50 and 51 in FIG. 3A disappear from the non-transmissive area 31 for displaying the surrounding image of the display device 11 in FIG. Conversely, the surrounding image 52 in FIG. 3A enters the non-transparent region 31 in FIG.

  In this way, the user 20 continues to move (scroll) the surrounding image displayed in the non-transparent area 31 while the transparent area 30 is in contact with or beyond the end face 41 of the window. It is possible to view an image outside the range of view (dead angle) from the entire surrounding image generated by.

  As described above, the visual field support device according to the first embodiment is based on a plurality of imaging devices (1 to 3) that capture images around a moving body and images captured by the plurality of imaging devices (1 to 3). Generated by the integrated image generation device 10 that synthesizes all-around images, the transmissive region 30 that can be worn on the user's head and can see the scene at the center of the field of view as seen from the user, and the integrated image generation device 10 A display device 11 having a non-transmissive region 31 surrounding the transmissive region 30 for displaying a part of the omnidirectional image, and a head motion detecting device 12 for detecting the movement of the head of the user 20. The display device 11 scrolls the entire surrounding image displayed in the non-transparent region 31 based on the movement information of the head of the user 20 detected by the head motion detection device 12. Process.

  In this way, by continuously scrolling the surrounding image displayed in the non-transmissive region 31, the user 20 sees an image outside the range of view (dead angle) from the entire surrounding image generated by the integrated image generating device 10. be able to. In this way, the user 20 has a head (an outside scene) in which an object that is within the field of view can be seen directly, and an image that is outside the user's field of view (dead angle) among all surrounding images generated by the integrated image. Since it can be simultaneously viewed by the display device mounted on the part, it is possible to see outside the field of view (dead angle) without moving the head, and it is possible to reduce the burden on the user's body, particularly the neck.

Embodiment 2. FIG.
FIG. 4 shows the movement of the image of the visual field support apparatus according to the second embodiment when the surrounding image is moved according to the movement of the eyeball of the user 20, and FIG. FIG. 4B is a diagram showing a state in which an external scene 53 and surrounding images 50 and 51 of the transmissive area 30 are visible before scrolling through a non-transmissive area 31 surrounding the transmissive area 30 and FIG. 11 is a diagram showing a state in which an external scene 53 and a surrounding image 52 of the transmissive area 30 appear to be scrolled through 11 transmissive areas 30 and a non-transmissive area 31 surrounding the transmissive area 30. The visual field support device according to the second embodiment is further provided with an eyeball motion detection device 13 that detects the movement of the eyeball of the user 20, and the eyeball motion detection device 13 is connected to the display device 11.

  The eyeball movement detection device 13 detects the movement of the eyeball of the user 20 and outputs the line-of-sight direction 60 to the display device 11 as movement information. In the eyeball motion detection device 13, an eyeball imaging device (camera) (not shown) installed outside the display device 11 images the eyeball of the user 20 through the transmission region 30 and performs image processing on the eyeball photographed image. Thus, the movement of the eyeball of the user 20 is detected. As a result, when it is recognized that the eye gaze direction 60 exists in the non-transmissive area 31, it is displayed in the non-transmissive area 31 from the direction facing the line-of-sight direction 60 toward the front (facing the transmissive area 30). The surrounding images 50 to 52 of the transparent area 30 are moved (scrolled). Further, when the eyeball motion detection device 13 detects from the movement of the eyeball of the user 20 that the line-of-sight direction 60 is reversed from the direction at the start of scrolling toward the front direction after the scrolling starts, the surrounding image Stop scrolling 50-52.

  As can be easily seen from FIG. 4, the display device 11 shown in FIG. 3 does not need to recognize the relationship between the transmission region 30 and the end face 41 of the window of the moving body, and the user 20 only changes the line-of-sight direction 60. It is possible to move (scroll) surrounding images outside the field of view (dead angle) displayed in the non-transmissive region 31 without moving the head. For this reason, the burden concerning a body, especially a neck part can be further reduced.

Embodiment 3 FIG.
FIG. 5 shows the movement of the image according to the third embodiment when the surrounding image is moved according to the movement of the cursor (pseudo gaze) 70 displayed in the non-transparent area 31. FIG. 5A shows a state in which an external scene 53 and surrounding images 50 and 51 of the transmissive area 30 can be seen before scrolling through the transmissive area 30 of the display device 11 and the non-transmissive area 31 surrounding the transmissive area 30. FIG. 5B shows a state in which the scenery 53 of the outside world and the surrounding image 52 of the transmissive area 30 appear to scroll through the transmissive area 30 of the display device 11 and the non-transmissive area 31 surrounding the transmissive area 30. FIG.

  In the display support device according to the third embodiment, the display device 11 is provided with a cursor display unit (not shown) that displays a cursor (pseudo gaze) 70 on the display screen of the display device 11. The cursor display unit displays a cursor (pseudo line of sight) 70 on the display screen of the display device 11. The cursor display unit moves the position of the cursor (pseudo gaze) 70 within the display screen of the display device 11 based on an operation input by the user 20 to the input instruction device (not shown) of the display device 11. be able to.

  Based on information other than the output of the eyeball motion detection device 13, the user 20 presents the display position of the cursor (pseudo gaze) 70 to the display device 11. When the display device 11 recognizes that the cursor (pseudo gaze) 70 is located in the non-transparent area 31 based on the display position of the cursor (pseudo gaze) 70, the display device 11 starts from the position where the cursor (pseudo gaze) 70 exists. The surrounding images 50 to 52 of the transmissive area 30 displayed in the non-transmissive area 31 are moved (scrolled) in the direction toward the transmissive area 30. Further, when the display device 11 detects that the cursor (pseudo line of sight) 70 has reversed from the direction at the start of scrolling to the front direction in accordance with the cursor operation instruction from the user 20 after scrolling has started. Then, the scrolling of the surrounding images 50 to 52 is stopped.

  According to FIG. 5, the head movement detection device 12 for detecting the movement of the head of the user 20 shown in FIG. 3, and the eye movement detection device 13 for detecting the movement of the eyeball of the user 20 shown in FIG. Is not necessary, and the movement (scrolling) of the surrounding image outside the viewing range (dead angle) displayed in the non-transparent area 31 does not depend on the movement of the user 20, so that it is controlled by a person other than the user 20. It becomes possible to do.

Embodiment 4 FIG.
FIG. 6 shows the configuration of the fourth embodiment in the case of optimizing the movement (scrolling) of surrounding images outside the field of view (dead angle) displayed in the non-transparent area 31 of the display device 11 for each user. . Specific information for each user is used, such as differences in the distance between the display device 11 attached to the head and the window 40 or the end face 41 of the moving body due to differences in the movement of the head or eyeballs or differences in physique. Stored in the person information storage device 14. According to FIG. 6, by transmitting unique information according to the user to the display device 11, the position of the line of sight and the wrinkle of movement can be added to the display control, and the display accuracy can be improved. It becomes possible.

  ADVANTAGE OF THE INVENTION According to this invention, the surrounding image which becomes out of the range of the visual field which a passenger | crew needs for a control, monitoring, etc. in moving bodies, such as an aircraft and a vehicle, can be seen. The present invention can be widely used in aircraft, ships, automobile industry, and transportation industry.

  1 to 3 imaging devices, 10 integrated image generation devices, 11 display devices, 12 head motion detection devices, 13 eyeball motion detection devices, 20 users, 14 user information storage devices, 30 transmission regions, 31 non-transmission regions, 40 Window of moving body, 41 End face of window, 50-52 surrounding image, 53 Scenery of the outside world, 60 user's gaze direction, 70 cursor (pseudo gaze).

Claims (4)

A plurality of imaging devices that capture images around the moving body;
An integrated image generation device that synthesizes an all-around image from images captured by the plurality of imaging devices;
A transmissive region that can be worn on the user's head and can see a scene at the center of the field of view as viewed from the user, and a transmissive region for displaying the all-around image generated by the integrated image generation device are surrounded. A display device having a non-transmissive region;
A head motion detection device for detecting the movement of the user's head;
With
The above-mentioned display device is a field-of-view assistance device that performs processing for scrolling the entire surrounding image displayed in the non-transparent area based on the movement information of the user's head detected by the head motion detection device. A plurality of imaging devices that capture images around the moving body;
An integrated image generation device that synthesizes an all-around image from images captured by the plurality of imaging devices;
A transmissive region that can be worn on the user's head and can see a scene at the center of the field of view as viewed from the user, and a transmissive region for displaying the all-around image generated by the integrated image generation device are surrounded. A display device having a non-transmissive region;
An eye movement detection device for detecting the movement of the eyeball of the user;
With
The visual field support device that performs a process of scrolling the all-around image displayed in the non-transparent area based on the movement information of the user's eyeball detected by the eyeball motion detection device. A plurality of imaging devices that capture images around the moving body;
An integrated image generation device that synthesizes an all-around image from images captured by the plurality of imaging devices;
A transmissive region that can be worn on the user's head and can see a scene at the center of the field of view as viewed from the user, and a transmissive region for displaying the all-around image generated by the integrated image generation device are surrounded. A display device having a non-transmissive region;
A cursor display unit that simulates the user's line of sight,
The above-mentioned display device is a field-of-view support device that performs processing for scrolling the entire surrounding image displayed in the non-transparent area based on the position information of the cursor detected by the cursor display unit.   The visual field support device according to any one of claims 1 to 3, further comprising a user information storage device that stores unique information for each user.

Images